DataPortal, Webservers, Large Files in DB

DataPortal, Webservers, Large Files in DB

Old forum URL: forums.lhotka.net/forums/t/4449.aspx


JoeFallon1 posted on Monday, March 03, 2008

Rocky,

If a large file (say 10MB or so) is stored in the database and you want to let a user download it in their browser, you can do this cleanly in ASP.Net 3.5 by using dr.SequentialAccess and reading chunks in buffers of 8040 bytes each. This kind of code naturally assumes the web server has a direct connection to the DB server. The nice part of this code is that the full file is not loaded into web server memory - it is retrieved in chunks, written out to the browser in the same size chunks and then repeated until the file is completely sent.

I can't see how this can be done with a DataPortal though. (Currently we limit file size to 1 MB to minimize this issue.)

Any ideas?

Joe

PS - actually the same problem occurs (in reverse) when writing the file to the DB. If a direct connection is allowed it is pretty straightforward. ASP.Net 3.5 allows the file to uploaded in 80 KB chunks which are then written to the file system! So the memory pressure on the web server is kept to a minimum. Older versions of ASP.Net forced the entire file into memory. A BinaryReader can then read the uploaded file as a stream and send it to the DB in chunks.

 

RockfordLhotka replied on Monday, March 03, 2008

The data portal doesn't support that sort of streaming, and I'm not sure it can.

However, WCF does support that sort of streaming - at least I've seen demos of streaming from Microsoft folks - streaming music, etc. It pretty much works the same way you describe, only at the service level rather than the database level.

So you could create another endpoint in your same app server virtual root (assuming you are using WCF for the data portal anyway) that exposes a streaming interface for upload/download of large file data.

Copyright (c) Marimer LLC