What scope is required to read an uploaded file

What scope is required to read an uploaded file

I am uploading a file to Drive using the JS client, then trying to access
that file from my server app by downloading its content. The server code
is ...
HttpResponse resp = service.getRequestFactory().buildGetRequest(new
GenericUrl(file.getDownloadUrl())).execute();
This works fine if the user has granted the all drive scope. However if
the user only grants drive.file, the call fails with a 404 error.
If I look at the uploaded file using the Drive web page, it shows as being
created by my app.
What do I need to do to read my uploaded file without drive scope?