Stable upload for large files

[denis]

For larges files, upload via http is not the most stable, perhaps it would be interesting to add the possibility to retrieve the video on a local ftp.

[vik]

This might be done by running a real ftp server (i.e. not the zope ftp server) which can handle resumes (zope's ftp apparently can't). The upload directory would be periodically checked for new files which would then put the file into plumi, upon which the user could add metadata.

[vik]

Another possible solution to the general problem (resume downloads) is to investigate a java upload applet - http itself should be able to resume uploads, but few clients are capable of this.

[vik]

I mean resume uploads

[vik]

More ideas/braindumping:

* User creates video item (i.e. fills in the metadata), then on the final stage has two options: either upload the video as we currently do it, or they get a short-term ftp login which will allow them to upload the file (and not much else) using an ftp client. When they have finished the upload, they click some button which then puts the uploaded file in the correct location, and begins transcoding etc.

* Use an ftp server with resume capability as above, but use a java applet in place of an ftp client. These exist, but I can't find any OSS ones.

* Use WebDAV. Zope contains a webdav server. If this allows for resuming uploads (the protocol allows this; individual clients/servers may or may not), then the user can access the plone site (or part thereof) as if it were part of the filesystem. Most recent operating systems (winXP, osx, linux) have support for webDAV.

[andycat]

see here , for latest FTP wrapper script. http://plumi.org/browser/ftpwrapper/trunk/zope_ftpd.py

This just needs a mechanism for the ftp wrapper to be able to initialise itself with the list of usernames/passwords from Plone, or be able to check via some mechanism (xmlrpc etc)

Other than that, this wrapper implements resumable FTP uploads.

[vik]

There is a similar discussion on the engagemedia trac:  https://trac.engagemedia.org/projects/ticket/521

I've remaned this to represent the more general case for large uploads, as it does not need to be specific to ftp.

I see an number of different possible solutions, with three possible protocols:

• ftp - protocol supports resuming
  • ftp service running on the server, either as part of plone or as a seperate service with plugin (already implemented to some degree). Requires the user to run a seperate piece of software (ftp client) to complete the upload, probably not an ideal scenario.
• http - protocol doesn't support resume (though there are proposals to make this happen:  http://code.google.com/p/gears/wiki/ResumableHttpRequestsProposal ). So the way to make reliable uploads via http is to split the file into smaller pieces on the client, and have the server stitch them back together
  • Flash or Java uploader: would require the user to have flash or java plugin installed. We can assume most users have flash, but it is a closed format, so not preferable. Java is common, but doesn't quite have the penetration of flash. Link to plugin for uploader may be an option (force people to use java), with fallback to non-reliable upload method.
  • Javascript uploader: Locally run javascript would handle splitting the file and sending it piece by piece. Firefox 3 has the capability to give JS access to local files via a form element. Other browsers may not - need to check this (particularly IE). Nice solution as it requires nothing except the browser. Code examples  http://www.finalcog.com/access-file-field-from-javascript and  http://igstan.blogspot.com/2009/01/pure-javascript-file-upload.html
  • Some other browser-based plugin: google gears probably provides this sort of functionality for uploading large files. Again with the reliance on browser extensions...
• Bittorrent: provides fault tolerant transfer mechanism
  • User seeds the file, the server leeches it. Would require extra piece of software on the user's side (except for opera which includes a BT client), and would need a tracker (could be the server). Creating and seeding the torrent may be complex for 'average user'; helper app might be needed. Disadvantage again of needing extra software. Advantage that if user continues to seed torrent it may provide for faster subsequent downloads in that user's area (need to investigate and re-ignite BT development in plumi).

I'm all in favour of simplicity as far as the user is concerned. So a browser based http solution would be ideal for me. At this point that would require the client (javascript or java or flash to split the local file) and server (capacity to receive split file, and re-construct) to be modified.

[vik]

resumable FTP uploads implemented as a wrapper for the zope FTP server: http://plumi.org/browser/ftpwrapper

Also available in the cheeseshop:  http://pypi.python.org/pypi/plumiftp/ which means it is installable via easy_install

[and]

This is in the 'ready for testing' state. Re-opening to fully resolve it.