Can I stream a file upload to S3 without a content-length header?

You have to upload your file in 5MiB+ chunks via S3's multipart API. Each of those chunks requires a Content-Length but you can avoid loading huge amounts of data (100MiB+) into memory.

Initiate S3 Multipart Upload.

Gather data into a buffer until that buffer reaches S3's lower chunk-size limit (5MiB). Generate MD5 checksum while building up the buffer.

Upload that buffer as a Part, store the ETag (read the docs on that one).

Once you reach EOF of your data, upload the last chunk (which can be smaller than 5MiB).

Finalize the Multipart Upload.

S3 allows up to 10,000 parts. So by choosing a part-size of 5MiB you will be able to upload dynamic files of up to 50GiB. Should be enough for most use-cases.

However: If you need more, you have to increase your part-size. Either by using a higher part-size (10MiB for example) or by increasing it during the upload.

First 25 parts:   5MiB (total:  125MiB)
Next 25 parts:   10MiB (total:  375MiB)
Next 25 parts:   25MiB (total:    1GiB)
Next 25 parts:   50MiB (total: 2.25GiB)
After that:     100MiB

This will allow you to upload files of up to 1TB (S3's limit for a single file is 5TB right now) without wasting memory unnecessarily.

A note on your link to Sean O'Donnells blog:

His problem is different from yours - he knows and uses the Content-Length before the upload. He wants to improve on this situation: Many libraries handle uploads by loading all data from a file into memory. In pseudo-code that would be something like this:

data = File.read(file_name)
request = new S3::PutFileRequest()
request.setHeader('Content-Length', data.size)
request.setBody(data)
request.send()

His solution does it by getting the Content-Length via the filesystem-API. He then streams the data from disk into the request-stream. In pseudo-code:

upload = new S3::PutFileRequestStream()
upload.writeHeader('Content-Length', File.getSize(file_name))
upload.flushHeader()

input = File.open(file_name, File::READONLY_FLAG)

while (data = input.read())
  input.write(data)
end

upload.flush()
upload.close()

---

Putting this answer here for others in case it helps:

If you don't know the length of the data you are streaming up to S3, you can use S3FileInfo and its OpenWrite() method to write arbitrary data into S3.

var fileInfo = new S3FileInfo(amazonS3Client, "MyBucket", "streamed-file.txt");

using (var outputStream = fileInfo.OpenWrite())
{
    using (var streamWriter = new StreamWriter(outputStream))
    {
        streamWriter.WriteLine("Hello world");
        // You can do as many writes as you want here
    }
}

---

You can use the gof3r command-line tool to just stream linux pipes:

$ tar -czf - <my_dir/> | gof3r put --bucket <s3_bucket> --key <s3_object>

---

If you are using Node.js you can use a plugin like s3-streaming-upload to accomplish this quite easily.

---

Refer more on HTTP multi-part enitity requests. You can send a file as chunks of data to the target.

---

reference to :https://github.com/aws/aws-cli/pull/903

Here is a synopsis: For uploading a stream from stdin to s3, use: aws s3 cp - s3://my-bucket/stream

For downloading an s3 object as a stdout stream, use: aws s3 cp s3://my-bucket/stream -

So for example, if I had the object s3://my-bucket/stream, I could run this command: aws s3 cp s3://my-bucket/stream - | aws s3 cp - s3://my-bucket/new-stream

my cmd:

echo "ccc" | aws --endpoint-url=http://172.22.222.245:80 --no-verify-ssl s3 cp - s3://test-bucket/ccc

---