StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

POStreaming a CSV file in Django
text
Body
copied!<p>I am attempting to stream a csv file as an attachment download. The CSV files are getting to be 4MB in size or more, and I need a way for the user to actively download the files without waiting for all of the data to be created and committed to memory first.</p> <p>I first used my own file wrapper based on Django's <code>FileWrapper</code> class. That failed. Then I saw a method here for using a generator to stream the response: <a href="https://stackoverflow.com/questions/2922874/how-to-stream-an-httpresponse-with-django">How to stream an HttpResponse with Django</a></p> <p>When I raise an error within the generator, I can see that I am creating the proper data with the <code>get_row_data()</code> function, but when I try to return the response it comes back empty. I've also disabled the Django <code>GZipMiddleware</code>. Does anyone know what I'm doing wrong?</p> <p><strong>Edit: The issue I was having was with the <code>ConditionalGetMiddleware</code>. I had to replace it, the code is in an answer below.</strong></p> <p>Here is the view:</p> <pre><code>from django.views.decorators.http import condition @condition(etag_func=None) def csv_view(request, app_label, model_name): """ Based on the filters in the query, return a csv file for the given model """ #Get the model model = models.get_model(app_label, model_name) #if there are filters in the query if request.method == 'GET': #if the query is not empty if request.META['QUERY_STRING'] != None: keyword_arg_dict = {} for key, value in request.GET.items(): #get the query filters keyword_arg_dict[str(key)] = str(value) #generate a list of row objects, based on the filters objects_list = model.objects.filter(**keyword_arg_dict) else: #get all the model's objects objects_list = model.objects.all() else: #get all the model's objects objects_list = model.objects.all() #create the reponse object with a csv mimetype response = HttpResponse( stream_response_generator(model, objects_list), mimetype='text/plain', ) response['Content-Disposition'] = "attachment; filename=foo.csv" return response </code></pre> <p>Here is the generator I use to stream the response:</p> <pre><code>def stream_response_generator(model, objects_list): """Streaming function to return data iteratively """ for row_item in objects_list: yield get_row_data(model, row_item) time.sleep(1) </code></pre> <p>And here is how I create the csv row data:</p> <pre><code>def get_row_data(model, row): """Get a row of csv data from an object""" #Create a temporary csv handle csv_handle = cStringIO.StringIO() #create the csv output object csv_output = csv.writer(csv_handle) value_list = [] for field in model._meta.fields: #if the field is a related field (ForeignKey, ManyToMany, OneToOne) if isinstance(field, RelatedField): #get the related model from the field object related_model = field.rel.to for key in row.__dict__.keys(): #find the field in the row that matches the related field if key.startswith(field.name): #Get the unicode version of the row in the related model, based on the id try: entry = related_model.objects.get( id__exact=int(row.__dict__[key]), ) except: pass else: value = entry.__unicode__().encode("utf-8") break #if it isn't a related field else: #get the value of the field if isinstance(row.__dict__[field.name], basestring): value = row.__dict__[field.name].encode("utf-8") else: value = row.__dict__[field.name] value_list.append(value) #add the row of csv values to the csv file csv_output.writerow(value_list) #Return the string value of the csv output return csv_handle.getvalue() </code></pre>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload