Databases
StackOverflow2013
Postsdbo
POAndroid most efficient way to read a large csv file and send DATA to server (Not uploading the file)
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POAndroid most efficient way to read a large csv file and send DATA to server (Not uploading the file)
    primarykey
    data
    text
    <p>I have a large csv file in sdcard and i want to know the best way to send the data from my csv file to my hosted server through my android app. My app writes to the csv file for an hour and then at the end of every hour i run a service which reads this csv file and sends it to the server.</p> <p>I was initially not writing anything to the csv file and sending each data as http post request. But it turned out that my app was showing too much of data usage for uploading. </p> <p>So i switched to writing the data to a file for an hour and at the end of the hour read each line and convert each tuple to json objects , put it into a json array and then send that array as a json object via name value pair.</p> <p>Problem 1: </p> <p>My CSV file is too large so I get <strong>Out Of Memory Exception</strong> when i try to add more than 2000 json objects to the array and send like this (Problem occurs on line marked with **)</p> <pre class="lang-java prettyprint-override"><code>nameValuePairs.add(new BasicNameValuePair("myjson", json_array_obj.toString())); Thread server = new Thread(){ String line = null; public void run() { try{ HttpClient httpclient1 = new DefaultHttpClient(); HttpPost httppost1 = new HttpPost("http://www.example.com/filename.php"); **httppost1.setEntity(new UrlEncodedFormEntity(nameValuePairs));** HttpResponse httpResponse = httpclient1.execute(httppost1); HttpEntity httpEntity = httpResponse.getEntity(); line = EntityUtils.toString(httpEntity); Log.e("line" , line); } catch(Exception e) { Log.v("catch", "executed"); } } }; server.start(); </code></pre> <p>Logcat showed :</p> <pre><code>`12-28 13:01:18.828: E/AndroidRuntime(2690FATAL EXCEPTION: Thread-16132 12-28 13:01:18.828: E/AndroidRuntime(26908): java.lang.OutOfMemoryError 12-28 13:01:18.828: E/AndroidRuntime(26908): at java.lang.AbstractStringBuilder.enlargeBuffer(AbstractStringBuilder.java:94) 12-28 13:01:18.828: E/AndroidRuntime(26908): at java.lang.AbstractStringBuilder.append0(AbstractStringBuilder.java:145) 12-28 13:01:18.828: E/AndroidRuntime(26908): at java.lang.StringBuilder.append(StringBuilder.java:216) 12-28 13:01:18.828: E/AndroidRuntime(26908): at libcore.net.UriCodec.appendHex(UriCodec.java:212) 12-28 13:01:18.828: E/AndroidRuntime(26908): at libcore.net.UriCodec.appendHex(UriCodec.java:206) 12-28 13:01:18.828: E/AndroidRuntime(26908): at libcore.net.UriCodec.appendEncoded(UriCodec.java:109) 12-28 13:01:18.828: E/AndroidRuntime(26908): at libcore.net.UriCodec.encode(UriCodec.java:133) 12-28 13:01:18.828: E/AndroidRuntime(26908): at java.net.URLEncoder.encode(URLEncoder.java:57) 12-28 13:01:18.828: E/AndroidRuntime(26908): at org.apache.http.client.utils.URLEncodedUtils.encode(URLEncodedUtils.java:184) 12-28 13:01:18.828: E/AndroidRuntime(26908): at org.apache.http.client.utils.URLEncodedUtils.format(URLEncodedUtils.java:163) 12-28 13:01:18.828: E/AndroidRuntime(26908): at org.apache.http.client.entity.UrlEncodedFormEntity.&lt;init&gt;(UrlEncodedFormEntity.java:71) 12-28 13:01:18.828: E/AndroidRuntime(26908): at com.example.acgylo.CSVtoServer$1.run(CSVtoServer.java:168)8): ` </code></pre> <p>I then found out that json array with large number of objects cannot be sent as namevaluepair as it will cause memory overflow.</p> <p>So i want to know what is the best way to do this :</p> <p>option 1 : Send data in batches of 500(i.e 500 lines of csv together).</p> <p>Is there any other efficient way to achieve sending of 2k tuples(or more) from CSV via HTTP in android ?</p> <p>Problem 2:</p> <p>a) I wanted to know how to reduce the data usage of android application ? </p> <p>b) What uses less data : </p> <pre><code>One http request with large json object sent as string OR Multiple http requests with smaller json objects? </code></pre> <p>Please help me find an efficient solution to the above problems. Any help is appreciated. </p> <p>Thanks in advance.</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USAndroid
    1. USAndroid
    plurals
    1. PL
      singulars
      1. LTLinked
    2. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POAndroid most efficient way to read a large csv file and send DATA to server (Not uploading the file)
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POAndroid most efficient way to read a large csv file and send DATA to server (Not uploading the file)
      1. This table or related slice is empty.
      1. VTApproveEditSuggestion
    1. COCan you modify the server side to accept a zip file?
      singulars
      1. POAndroid most efficient way to read a large csv file and send DATA to server (Not uploading the file)
      1. USMorrison Chang
    2. COThere is a small amount of overhead in each request, along with the (slight) protocol overhead of sending JSON, which means that simply breaking up the larger request into smaller ones will likely increase your bandwidth use, not reduce it. The OOM is on your client side - can you not increase the memory available to it?
      singulars
      1. POAndroid most efficient way to read a large csv file and send DATA to server (Not uploading the file)
      1. USPerception
    3. CONo, i cant as it is a php file which inserts each tuple into the database
      singulars
      1. POAndroid most efficient way to read a large csv file and send DATA to server (Not uploading the file)
      1. USAndroid
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload