large time on wa (waiting io)

[yurj]

Hi!

using plumi 0.2:

more /var/lib/plone25/zeocluster/Products/ATVideo/version.txt 0.2 build 17 more /var/lib/plone25/zeocluster/Products/ExternalStorage/version.txt 0.7 (SVN/Unreleased) more /var/lib/plone25/zeocluster/Products/ATMediaFile/version.txt 0.1 build 66

it is very slow on shoing things. Using top, I can see the cpu 100% on wa (waiting io).

Can this be a problem of externalstorage?

[anonymous]

can be SearchableText?? I've uploaded 1 GB of videos, and reindexing SearchableText? take 30 seconds. Archetypes does a lot of reindexing, can be the guilt?

[yurj]

class VideoView?( BrowserView? ):

u"""This browser view is used as utility for the atvideo view """ implements( IVideoView, ITopicsProvider )

def init(self, context, request):

super(VideoView?, self).init(context, request) self.portal_url = getToolByName(self.context, "portal_url")() self.vocab_tool = getToolByName(self.context, "portal_vocabularies") self.use_vpip = "vpip" in context.Subject()

media_info = context.getFileAttribs() self.enclosure = media_info[1] > 0

(self.transcoding_ready,

self.transcoding_status) = self.get_transcoding_status()

is this ok? it is writing every time I see an item.

Also getFileAttribs is called two or three times on every view:

---

2008-07-24T17:50:40 INFO getFileAttribs using the CACHE of file system , length 6342048

---

2008-07-24T17:50:43 INFO getFileAttribs using the CACHE of file system , length 6342048

---

2008-07-24T17:50:45 INFO getFileAttribs using the CACHE of file system , length 6342048

ATVideo/skins/ATEngageVideo/video_view.pt: tal:define="enc here/getFileAttribs" ATVideo/skins/ATEngageVideo/video_view.pt: tal:define="enc here/getFileAttribs ; bt_url here/getTorrentURL" ATVideo/browser/video.py: media_info = context.getFileAttribs()

[monossido]

Seems i have the same issue, with a 400mb of video the page takes a lot of seconds to appear and 100% of cpu Here http://plumi.org/ticket/102#comment:3 i read "yes, this is a known issue with large file support inside Plone.

You should upgraded to Plone 2.5.4-2"

anyone have test this?

i'm italian so sorry for my bad english ;)

[yurj]

no, it does'nt change.

Maybe this is the real problem!

http://dev.plone.org/plone/ticket/7586

"""FileField?.getFilename by default calls getBaseUnit to get the filename. getBaseUnit recalls FileField?.getFilename with fromBaseUnit to false to get the filename ! After, it creates an BaseUnit? and loads the file content into memory (value.data). With big files, it's very slow and consummes a lot of RAM for nothing. After all this work, BaseUnit?.getFilename is called.

This is very complex and unefficient ! Why FileField?.getFilename needs to create a baseUnit only to get a filename that is got finally from itself ? """

[andycat]

I might close this ticket *after* we move to Blob fields, and if we dont get any more comments/feedback.

[yurj]

it has nothing to do with the storage. It is the standard template loading data from the content in the wrong way, just take a look to it with a big file.

takes 5 minute to test it.

[andycat]

narky comments about the length of time something takes to test arent helpful. Fact is : this bug report isnt very good quality, its very hard to reproduce whatever problem it is you are trying to describe.

[anonymous]

good quality:

install Plumi (0.2 oe 0.3, no differences about this topic) upload a 200MB video, wait the transcoding in flash view the page with the video (watch the event.log with tail -f to see messages about sizes and caching it)

It takes a lot of seconds to just render the view (doing the same thing three time, as you can see in the event.log)

Try it, takes few minutes.

[yurj]

simpler, go to: http://studidirittopenale.cab.unipd.it/ and try to see some videos.

[andycat]

So a video view is taking longer that you like ? How slow do you define too slow? Have you cached your site using apache2/squid ?

You say "It is the standard template loading data from the content in the wrong way, just take a look to it with a big file."

What is the template doing wrong?

As my comment above was alluding, once we move into Plone 3 and ATBlob fields, the speed of ZODB operations should be quicker, since the large binary objects (video) will be handled better.

[yurj]

Yes, I've cached but that content (the video object) send a nice (...) "please, don't cache me", if you look at the headers you'll see it. Even CacheFu? can't override this behaviour, I can't find where to look deeper...

(i'm using apache cache on the virtual host:

CacheRoot? /var/cache/www/dirittopenale CacheEnable? disk /

)

If you see, the home page load fast, being it cached by apache.

Anyway, the storage will not change anything, somewhere the video is loaded on memory or the code is doing some operation that load the video in memory, maybe deep in Plone or Archetypes or Zope.

I just ask you to try to see, quickly, why it takes so long. The flash part should not take long, it is just text. I suppose it is some code calculating the video size, and as side effect, loading it (or most) in memory.

You can't see it with small videos, you've to load a about 100 MB file (and more) to see it clearly.

I'm happy with Plumi, this is the only real issue I've found :)