I would like to know whether it is possible for wget when writing its output to
files to decrease its hard disk accesses while downloading.
in fact when i want to download a repertory on an ftp server which contains
thousands of files with a total size of a couple of GB with the following
command line:
$wget -r ftp://server/name -o output.log
, wget will append information to output.log after completing each file in the
repertory that i'm downloading and eventual midcourse information, so wget will
access thousand times to hard disk. I would prefer an option that allows it to
write the file only into ram and then write to hard disk only at the end.


--
heavytull

heavytull wrote:
I would prefer an option that allows it to
> write the file only into ram and then write to hard disk only at the end.
>
>
How much memory do you have? If you want only 1 write to hard drive you
will need to know the size of the download so you can make the RAM disk.

You might try recompiling 'wget' with the logging function writing to a
NULL device.

Or you might be creative and bold and look at the '-o' flag. Doesn't it
redirect the log output, this could be a NULL device.

--
A little rum in the morning coffee. Just to clear the cobwebs, ya know.

On 2007-07-07, heavytull wrote:
> I would like to know whether it is possible for wget when writing its output to
> files to decrease its hard disk accesses while downloading.
> in fact when i want to download a repertory on an ftp server which contains
> thousands of files with a total size of a couple of GB with the following
> command line:
> $wget -r ftp://server/name -o output.log
> , wget will append information to output.log after completing each file in the
> repertory that i'm downloading and eventual midcourse information, so wget will
> access thousand times to hard disk. I would prefer an option that allows it to
> write the file only into ram and then write to hard disk only at the end.

Let your operating system do its job. You are unlikely to come up
with anything more efficient.

--
Chris F.A. Johnson, author | <http://cfaj.freeshell.org>
Shell Scripting Recipes: | My code in this post, if any,
A Problem-Solution Approach | is released under the
2005, Apress | GNU General Public Licence

On Sat, 07 Jul 2007 23:43:48 +0200, heavytull wrote:

> I would like to know whether it is possible for wget when writing its output to
> files to decrease its hard disk accesses while downloading.
> in fact when i want to download a repertory on an ftp server which contains
> thousands of files with a total size of a couple of GB with the following
> command line:
> $wget -r ftp://server/name -o output.log
> , wget will append information to output.log after completing each file in the
> repertory that i'm downloading and eventual midcourse information, so wget will
> access thousand times to hard disk. I would prefer an option that allows it to
> write the file only into ram and then write to hard disk only at the end.
>
>

Maybe the shell can handle it better with a redirect.

wget -r ftp://server/name 2> output.log


stonerfish
--
Just say "I don't know"

heavytull wrote:

> $wget -r ftp://server/name -o output.log
> , wget will append information to output.log after completing each file in the
> repertory that i'm downloading and eventual midcourse information, so wget will
> access thousand times to hard disk. I would prefer an option that allows it to
> write the file only into ram and then write to hard disk only at the end.

The only way you can lessen the hard drive access is to store the log file to
ram (create large enough ramfs), when finished move it to your hard drive.

If you use konsole (part of KDE), you could set it's buffer to be large enough
to hold the output from wget, and all files fetched, save the buffer to hard
drive.

In both these cases you will need enough ram to store this, if you don't, then
in the first case you won't get the whole log and in the second case things
will be flushed into the swap (disk activity).


The built in log file feature in wget is better than using the shells redirect.

--

//Aho

J.O. Aho wrote:

> heavytull wrote:
>
>> $wget -r ftp://server/name -o output.log
>> , wget will append information to output.log after completing each file in
>> the repertory that i'm downloading and eventual midcourse information, so
>> wget will access thousand times to hard disk. I would prefer an option that
>> allows it to write the file only into ram and then write to hard disk only at
>> the end.
>
> The only way you can lessen the hard drive access is to store the log file to
> ram (create large enough ramfs), when finished move it to your hard drive.
>
yes this option interests me

> If you use konsole (part of KDE), you could set it's buffer to be large enough
> to hold the output from wget, and all files fetched, save the buffer to hard
> drive.
>
> In both these cases you will need enough ram to store this, if you don't, then
> in the first case you won't get the whole log and in the second case things
> will be flushed into the swap (disk activity).
>
>
> The built in log file feature in wget is better than using the shells
> redirect.
>

--
heavytull

After takin' a swig o' grog, Chris F.A. Johnson belched out this bit o' wisdom:

>> I would like to know whether it is possible for wget when writing its output to
>> files to decrease its hard disk accesses while downloading.
>
> Let your operating system do its job. You are unlikely to come up
> with anything more efficient.

Especially with Linux.

--
Tux rox!

heavytull wrote:
> J.O. Aho wrote:
>
>> heavytull wrote:
>>
>>> $wget -r ftp://server/name -o output.log
>>> , wget will append information to output.log after completing each file in
>>> the repertory that i'm downloading and eventual midcourse information, so
>>> wget will access thousand times to hard disk. I would prefer an option that
>>> allows it to write the file only into ram and then write to hard disk only at
>>> the end.
>> The only way you can lessen the hard drive access is to store the log file to
>> ram (create large enough ramfs), when finished move it to your hard drive.
>>
> yes this option interests me

mkdir -p /mnt/ram
mount /dev/ram1 /mnt/ram

And it's just to use, it's not large (size set in kernel during configuration).


--

//Aho

On Jul 8, 4:42 pm, Linonut <lino...@bellsouth.net> wrote:
> After takin' a swig o' grog, Chris F.A. Johnson belched out this bit o' wisdom:
>
> >> I would like to know whether it is possible for wget when writing its output to
> >> files to decrease its hard disk accesses while downloading.
>
> > Let your operating system do its job. You are unlikely to come up
> > with anything more efficient.
>
> Especially with Linux.
>
> --
> Tux rox!

I'm wondering whether the linux stiff (the kernel and filesystems
ext2, 3 and reiserfs) is tuned to save hardware lifespan.

hints:
1- just listen the noise difference that makes the hard drive while
writing or reading a lot of data under linux and then under windows
with the ntfs.

2- i've read that during years there was a cdrecording driver (the
default one for most distros) (for packet writing i think) that was
doing a bloody write test at the same address everytime a cdrw was
mounted under linux.
i mount cd's maybe 5 times a day, a cdrw is limited to 1000 write and
erase cycles. just count how would survive the cdrw.
i heard that pressure was exerced on developpers, maybe now it's ok.

3- i doubt whether usb-storage driver doesn't make a write test if
mounted a usk key that is not write protected or that doesn't have
that feature.

heavytull wrote:
> On Jul 8, 4:42 pm, Linonut <lino...@bellsouth.net> wrote:
>> After takin' a swig o' grog, Chris F.A. Johnson belched out this bit o' wisdom:
>>
>>>> I would like to know whether it is possible for wget when writing its output to
>>>> files to decrease its hard disk accesses while downloading.
>>> Let your operating system do its job. You are unlikely to come up
>>> with anything more efficient.
>> Especially with Linux.
>>
>> --
>> Tux rox!
>
> I'm wondering whether the linux stiff (the kernel and filesystems
> ext2, 3 and reiserfs) is tuned to save hardware lifespan.
>
> hints:
> 1- just listen the noise difference that makes the hard drive while
> writing or reading a lot of data under linux and then under windows
> with the ntfs.

Yeah, I agree, the windows box is noisy.


> 2- i've read that during years there was a cdrecording driver (the
> default one for most distros) (for packet writing i think) that was
> doing a bloody write test at the same address everytime a cdrw was
> mounted under linux.

Never seen anything like that and I really doubt that, as there is no reason
to make a write test to a block device at mount time.


> 3- i doubt whether usb-storage driver doesn't make a write test if
> mounted a usk key that is not write protected or that doesn't have
> that feature.

Why would it make a write test? A read test is more sensible.


--

//Aho

"J.O. Aho" <user@example.net> wrote in
news:5fcc49F3bh0d5U1@mid.individual.net:

>
> mkdir -p /mnt/ram
> mount /dev/ram1 /mnt/ram
>
> And it's just to use, it's not large (size set in kernel during
> configuration).
>

[root@ohmster mnt]# mkdir -p /mnt/ram
[root@ohmster mnt]# mount /dev/ram1 /mnt/ram
mount: you must specify the filesystem type
[root@ohmster mnt]#

It don't work Aho.

--
~Ohmster
ohmster at ohmster dot com
Put "messageforohmster" in message body
(That is Message Body, not Subject!)
to pass my spam filter.

> [root@ohmster mnt]# mkdir -p /mnt/ram
> [root@ohmster mnt]# mount /dev/ram1 /mnt/ram
> mount: you must specify the filesystem type
> [root@ohmster mnt]#
>
> It don't work Aho.
>

Cousin Ohm ....

Try giving mount a filesystem type ....

sudo mount -t ramfs /dev/ram1 /mnt/ram


--
Stanley C. Kitching
Human Being
Phoenix, Arizona


----== Posted via Newsfeeds.Com - Unlimited-Unrestricted-Secure Usenet News==----
http://www.newsfeeds.com The #1 Newsgroup Service in the World! 120,000+ Newsgroups
----= East and West-Coast Server Farms - Total Privacy via Encryption =----

Cousin Stanley <cousinstanley********.com> wrote in news:1184051730_69381
@sp6iad.superfeed.net:

>> [root@ohmster mnt]# mkdir -p /mnt/ram
>> [root@ohmster mnt]# mount /dev/ram1 /mnt/ram
>> mount: you must specify the filesystem type
>> [root@ohmster mnt]#
>>
>> It don't work Aho.
>>
>
> Cousin Ohm ....
>
> Try giving mount a filesystem type ....
>
> sudo mount -t ramfs /dev/ram1 /mnt/ram
>

Yes that worked quite well Cousin Stanley. :)

Thank you.

--
~Ohmster
ohmster at ohmster dot com
Put "messageforohmster" in message body
(That is Message Body, not Subject!)
to pass my spam filter.

After takin' a swig o' grog, heavytull belched out this bit o' wisdom:

> 2- i've read that during years there was a cdrecording driver (the
> default one for most distros) (for packet writing i think) that was
> doing a bloody write test at the same address everytime a cdrw was
> mounted under linux.

Good ol' Jörg!

--
Tux rox!

Cousin Stanley <cousinstanley********.com> wrote in news:1184051730_69381
@sp6iad.superfeed.net:

> Cousin Ohm ....
>
> Try giving mount a filesystem type ....
>
> sudo mount -t ramfs /dev/ram1 /mnt/ram
>

That did work Cousin Stanley,l or anyone, but how do you know the capacity
of the new ramdrive? There seems to be no way of determining that. Not even
df shows how much you can put in the new /mnt/ram/ mount.

--
~Ohmster
ohmster at ohmster dot com
Put "messageforohmster" in message body
(That is Message Body, not Subject!)
to pass my spam filter.