[Scribus] Memory problems when document contains many large pictures

[unknown]

My experience shows that scribus shouldn't be used for many large pictures.
Opening a file can take hours, often ended by a crash.

Observing the memory consumption gives hints to the underlying problem:
- during load all computer memory and also the swap partition is filled
- probably this is used to store all pictures in bitmap format
- reason is probably faster rescale when pictures are in memory
- when I set the images preview to "low resolution", memory is freed

Digital cameras produce very large pictures. It is common to have resulting
image resolution above 800 DPI.
On screen these pictures are very sharp even in "low resolution" mode.
These images consume lots of memory and slow down many operation of scribus.

My recommendations:

1) make image resolution for preview dependent on resulting DPI


2) give user a chance to load documents having too many large images. E.g.
load-dialog option "very low resolution".
Currently the image resolution after load is always "normal", independent of
the three possible settings:
- Setting all images to low resolution
- Document Settings - Tools - Images preview: low resolution
- Preferences - Tools - Images preview: low resolution

When producing output, the full resolution must be used.
I'm not sure whether recommendation 1) and 2) really helps to reduce memory
consumption...


3) separate the image loading into an asynchronous, lower priority thread.
When this thread has loaded all images of the current page it should load
near images during say 60 seconds - then finish.
So it would not load more images than used (adapted to computer speed and
image size). Many scribus operations would be much faster. On a dual-core
CPU the second core would load the images !

greetings
Stefan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://nashi.altmuehlnet.de/pipermail/scribus/attachments/20080111/bb8f1c5f/attachment.htm

[unknown]

On Fri, 11 Jan 2008 22:40:03 +0100

Post by unknown
My experience shows that scribus shouldn't be used for many large pictures.
Opening a file can take hours, often ended by a crash.
2) give user a chance to load documents having too many large images. E.g.
load-dialog option "very low resolution".

If it is a problem, why not untick 'Image visible' ?



Owen

[unknown]

Post by unknown
My experience shows that scribus shouldn't be used for many large
pictures. Opening a file can take hours, often ended by a crash.

That's not been my experience. What spec machine are you using ?

Post by unknown
- during load all computer memory and also the swap partition is filled
- probably this is used to store all pictures in bitmap format

I don't know how images are stored internally. We use lots of images
from 3 and 5 megapixel cameras, but I am using a 1.8 GHz P4 laptop with
2 gig of RAM, documents are typically 30 pages, with an average of one
or two images per page. Images are .jpg, either direct from the camera,
or sometimes messed with in Gimp.

Are you really running out of memory ? Do you have a limit on the swap
file size ? Is it possible that you have some underlying hardware
problem that only manifests when you use a lot of VM ? I seldom see
Scribus use more than 300 meg which doesn't seem to bother this system
at all.


Cheers, J/.

[unknown]

Owen wrote: If it is a problem, why not untick 'Image visible' ?

It does not change load time or memory consumption ...

-----

Post by unknown
I don't know how images are stored internally. We use lots of images
from 3 and 5 megapixel cameras, but I am using a 1.8 GHz P4 laptop with
2 gig of RAM, documents are typically 30 pages, with an average of one
or two images per page. Images are .jpg, either direct from the camera,
or sometimes messed with in Gimp.
Are you really running out of memory ? Do you have a limit on the swap
file size ? Is it possible that you have some underlying hardware
problem that only manifests when you use a lot of VM ? I seldom see
Scribus use more than 300 meg which doesn't seem to bother this system
at all.

With 2GB of memory, you will experience the problems when you have more than
about 300 images.
When your camera would have 20 Megapixel you would already have problems.

I only have 500MB memory + 3GB swap space on a linux laptop.
I also can work with documents up to about 30 pages (16 pages already take
12 minutes to load!).
But in the end my picture-book should have 500 images and 160 pages.
So I had to break it up and concat the different PDF's (with Adobe Acrobat).
It will probably not be possible to produce this even on your machine - at
least not with "professional" speed.

Stefan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://nashi.altmuehlnet.de/pipermail/scribus/attachments/20080112/5b57ab77/attachment-0001.htm

[unknown]

Post by unknown
Owen wrote: If it is a problem, why not untick 'Image visible' ?
It does not change load time or memory consumption ...
-----
I don't know how images are stored internally. We use lots of images
from 3 and 5 megapixel cameras, but I am using a 1.8 GHz P4 laptop with
2 gig of RAM, documents are typically 30 pages, with an average of one
or two images per page. Images are .jpg, either direct from the camera,
or sometimes messed with in Gimp.
Are you really running out of memory ? Do you have a limit on the swap
file size ? Is it possible that you have some underlying hardware
problem that only manifests when you use a lot of VM ? I seldom see
Scribus use more than 300 meg which doesn't seem to bother this system
at all.
With 2GB of memory, you will experience the problems when you have more
than about 300 images.

Maybe, but I wouldn't be surprised if that was true of most WP and DTP
software.

I'm not by any means suggesting that Scribus shouldn't be written to
make better / best use of memory, but can you get a quart out of a pint
pot ?

Post by unknown
When your camera would have 20 Megapixel you would already have problems.

If my camera had 20 megapixels I would expect them. This isn't so
different from a 300 dpi scanned A4 page, which I would expect to slow
things down if there were a lot of them.

Post by unknown
I only have 500MB memory + 3GB swap space on a linux laptop.
I also can work with documents up to about 30 pages (16 pages already
take 12 minutes to load!).

Hmmm... Back in the days of pagemaker I used to see silly delays
loading documents and moving to pages, but not that silly.
Circumstances were very different then though, on a P/200 with a massive
192 meg of RAM, and windows NT's VM. Most of our images we only one
megapixel then though, but some 300 dpi scanned pages weren't unusual.

Post by unknown
But in the end my picture-book should have 500 images and 160 pages.
So I had to break it up and concat the different PDF's (with Adobe Acrobat).
It will probably not be possible to produce this even on your machine -
at least not with "professional" speed.

Possibly, but it depends on what you call professional I expect. It's
interesting that upgrading this machine by throwing away the old simms
and buying two new 1 gig sticks only cost 43 UKP, so if you can afford
it, the 'professional' approach is probably to throw memory at it. This
said, I was doing OK in 1 gig if there wasn't much else running.

In the mean time, let's hope that the behaviour of Scribus can be made
optimal for the users of both large and small memory machines. I'd hate
to think that the users of larger machines couldn't fully exploit them
because of work arounds for small systems.


Cheers, J/.

[unknown]

Post by unknown
With 2GB of memory, you will experience the problems when you have
more than about 300 images.
When your camera would have 20 Megapixel you would already have problems.
I only have 500MB memory + 3GB swap space on a linux laptop.
I also can work with documents up to about 30 pages (16 pages already
take 12 minutes to load!).
But in the end my picture-book should have 500 images and 160 pages.
So I had to break it up and concat the different PDF's (with Adobe Acrobat).
It will probably not be possible to produce this even on your machine
- at least not with "professional" speed.

Hi, can you please give a full hardware/software detail on wich PC are
you running scribus?

I'm using it on an:
AMD Athlon (Barton) 3000+ (2200MHz)
1Gb PC3200 400MHz DDR-SDRAM (2x512Mb Kingston ValueRAM)
80Gb SATA Hard Drive
MSI KT6 mainboard (MS-6712)
Generic Nvidia 6600GT AGP8x 256Mb Video Card.

You can see i have not a current generation computer.

Please, note that something as simple as using a different motherboard
on MY system can make it go in a performance loss of 60% (that is a 3 to
1 relation)
I'm a Hardware specialist and i can assure (ensure?) you that
performance difference between motherboards can go as far as 6 to 1 loss
(or even worst).

I am currently using Scribus 1.3.3.10svn as fast as 1.3.3.9 and making
16 pages (8 to 16 pictures/page) catalogs wich open in less than 1 minute.
I?m using pictures in png format (as i couldn't use transparency from
.gif imgs), several svg imported logos, text frames, picture frames
modifyed for text surrounding pictures and i can't see a real
performance issue yet.

See that 500Mb RAM is not a correct memory ammount, you may be using 504
or 496 mb, wich tells me you are using an 8 or 16Mb onboard videocard,
wich lets you on a very old computer and 8/16Mb is far less that needed
for managing images the size you are, perhaps an M805/810/825/840
Pcchips mainboard (or similar) with an athlon or duron CPU or equivalent
mainboard for PIV. Maybe a PIII ?..

Can you please give that detail?...
Thanks.

Post by unknown
Stefan
------------------------------------------------------------------------
_______________________________________________
Scribus mailing list
Scribus at nashi.altmuehlnet.de
http://nashi.altmuehlnet.de/mailman/listinfo/scribus

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://nashi.altmuehlnet.de/pipermail/scribus/attachments/20080112/746fef59/attachment.htm

[unknown]

Hi, can you please give a full hardware/software detail on which PC are
you running scribus?

It's just my Linux test laptop, a DELL Latitude C810, 503.7MB RAM, Pentium
III Mobile, 1133MHz, Ubuntu Linux 7.10.
I don't want to upgrade this old computer and did not expect scribus to be
very fast on it.

My intention is to tell, that any software that loads all pictures as
bitmaps into memory will get very slow when either you have a slow machine,
a large book or many large pictures.
Images should only be loaded when needed, this would scale much better.

cheers
Stefan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://nashi.altmuehlnet.de/pipermail/scribus/attachments/20080112/7ae973f4/attachment-0001.htm

[unknown]

Post by unknown
Images should only be loaded when needed, this would scale much better.

:-)

the asynchronous proposal and the options proposal sound nice too.

JL

[unknown]

Post by unknown
My experience shows that scribus shouldn't be used for many large pictures.
Opening a file can take hours, often ended by a crash.

It's a known issue that's been around for a while, but it really should
absurdly huge images to trigger. Memory problems are more common during
PDF export, but some improvements have been made there too. At one point
you used to need at least four times the uncompressed size of the image
to export it to PDF, but I think that's been reduced significantly.

There was some discussion of progressively loading, processing and
converting/resampling/etc images rather than loading the whole lot into
RAM at once. I don't know how much if any of that sort of thing has made
it into the code, but but it's probably the best way to handle image
memory issues eventually, both for export and for creating display previews.

A bug in Scribus (in image processing or elsewhere) could also cause it
to allocate and use a huge amount of RAM that'd lead to a crash, so the
issue isn't necessarily caused by normal image handling.

Post by unknown
- during load all computer memory and also the swap partition is filled

Out of interest, how much RAM do you have?

Post by unknown
3) separate the image loading into an asynchronous, lower priority thread.
When this thread has loaded all images of the current page it should load
near images during say 60 seconds - then finish.

This sort of thing would be needed to help Scribus support longer
documents better, and it's a good idea in the long run.

--
Craig Ringer