Databases
StackOverflow2013
Postsdbo
POCan anyone explain the socket throughput discrepancies experienced between Windows (7 Pro N) and Linux ( Ubuntu 12-04)
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POCan anyone explain the socket throughput discrepancies experienced between Windows (7 Pro N) and Linux ( Ubuntu 12-04)
    primarykey
    data
    text
    <p>While evaluating 3rd party software (Java framework utilising NIO) we discovered that the throughput of the framework on Windows was about 50% of that on Linux. </p> <p>Assuming that the there are some JVM or OS settings affecting Windows we set about to test simple computational (Fibonacci, heapsort, strcat, etc.) and object building across both platforms. In all cases the OSs are pretty much even. However when we performed a throughput test using a simple ServerSocket and Client Socket (java.net jdk 1.7u5) we noticed that the Linux throughput can be as high as 10 times that of Windows especially for small messages (100 bytes).</p> <p>Our immediate assumption was that the OS socket send/receive buffer sizes are different and that they are playing a significant role in the throughput. Or that is related to TcpNoDelay being enabled on one OS and not the other. </p> <p>OS Socket Option defaults are ...</p> <pre><code>Linux: rcvBuf: 43690, sndBuf:86700, TcpNoDelay: No (Nagle On) Windows7: rcvBuf: 8192, sndBuf:8192, TcpNoDelay: No (Nagle On) </code></pre> <p>The results are as follows ...</p> <p><img src="https://i.stack.imgur.com/mAv5a.png" alt="Throughput Test Results"></p> <p>There are many questions that arise from these results though the following are the most challenging (I simply can’t explain) and the most immediate as the current target platform is Windows.</p> <ol> <li><p>Why are Windows times for 100 byte messages (with Nagle On) regardless of buffer sizes almost 10 times slower than Linux</p></li> <li><p>Why are all Windows times for 100 byte messages (regardless of Nagle setting or buffer sizes) roughly 7000ms</p></li> <li><p>Why are all Windows time s the 100 byte messages (regardless of Nagle setting or buffer sizes) slower than their 1000 byte equivalent. I suspect this is related to efficient use of buffering and/or Nagling On/Off though I don’t fully understand it.</p></li> </ol> <p>We performed additional tests on Windows where we start the message size at 50 bytes and increment 100 bytes after each 1 million messages. The results are strange (and yet predictable); between 50 and 400-450 bytes the times are ~7000ms, from 450-1000 bytes they range from ~5500ms - ~7000ms. Repeating this test over and over shows the same pattern. It’s like the small messages require to be padded out to a specific size (perhaps the MTU or other constraint). Again we are unable to explain this behaviour.</p> <p>Any explanations or direction would be most appreciated. Please refrain from reducing this to a Windows v Linux topic, keep that for another day.</p> <p><strong>Server Code</strong></p> <pre><code>public class SimpleMultiClientServer implements Runnable { private static ExecutorService threadPool = null; public static void main(String[] args) throws IOException { //ExecutorService threadPool; AppConfiguration configurations = null; if ((configurations = AppConfiguration.readAppConfiguration(args)) == null ) System.exit(1); //create thread pool and execute threadPool = Executors.newCachedThreadPool(); threadPool.execute( new SimpleMultiClientServer(configurations) ); System.out.println("Hit any key to exit ...."); System.in.read(); //stop listening threadPool.shutdown(); } //------------------------------------------------------------------------------ private ServerSocket theServerSocket; private boolean listening = true; private AppConfiguration theConfigurations; private SimpleMultiClientServer(AppConfiguration configurations) { this.theConfigurations = configurations; } @Override public void run() { try { theServerSocket = new ServerSocket(theConfigurations.port); System.out.println(String.format("ServerSocket listening on port [%d]", theConfigurations.port)); if ( theConfigurations.printSocketDetails ) printSocketDetials(); setServerSocketConditioning(); } catch (IOException e) { System.err.println(String.format("Could not listen on port: %d.", theConfigurations.port)); System.exit(-1); } //TODO interrupt the listending thread in order to shutdown while (listening) { try { threadPool.execute( new SimpleMultiClientServerThread( theServerSocket.accept() ) ); System.out.println("Accept new client on socket ...."); } catch (IOException e) { // TODO Auto-generated catch block e.printStackTrace(); } } try { theServerSocket.close(); } catch (IOException e) { e.printStackTrace(); } } private static class SimpleMultiClientServerThread implements Runnable { private Socket theSocket = null; @SuppressWarnings("unused") private long messageCount = 0; public SimpleMultiClientServerThread(Socket socket) { this.theSocket = socket; } public void run() { PrintWriter out = null; BufferedReader in = null; try { out = new PrintWriter(theSocket.getOutputStream(), true); in = new BufferedReader(new InputStreamReader( theSocket.getInputStream() )); @SuppressWarnings("unused") String inputLine; while ((inputLine = in.readLine()) != null) { messageCount++; //TODO client should send shutdown message or end of stream marker on closing } } catch (IOException e) { e.printStackTrace(); } finally { try { if ( null != out ) out.close(); if ( null != in ) in.close(); if ( null != theSocket ) theSocket.close(); } catch (Exception e) { e.printStackTrace(); } } } } private void printSocketDetials() { //write to file FileOutputStream fos = null; PrintWriter pw = null; try { fos = new FileOutputStream("socketserver-details.log", true); pw = new PrintWriter(fos); pw.println("Socket (Server) details ...."); try { if ( null != this.theServerSocket ) { pw.println(String.format("isBound [%b]", this.theServerSocket.isBound())); //SO_RCVBUF pw.println(String.format("getReceiveBufferSize [%d]", this.theServerSocket.getReceiveBufferSize())); //SO_REUSEADDR pw.println(String.format("getReuseAddress [%b]", this.theServerSocket.getReuseAddress())); //SO_TIMEOUT pw.println(String.format("getSoTimeout [%d]", this.theServerSocket.getSoTimeout())); pw.println("----------------------------------------------------------"); } } catch (Exception e) { e.printStackTrace(); } System.out.println(); } catch (FileNotFoundException e) { // TODO Auto-generated catch block e.printStackTrace(); } finally { if ( null != pw ) { pw.flush(); pw.close(); if ( null != fos ) pw.close(); } } } private void setServerSocketConditioning() { try { if ( theConfigurations.rcvBufSpecified ) theServerSocket.setReceiveBufferSize(theConfigurations.rcvBuf); if ( ( theConfigurations.rcvBufSpecified ) &amp;&amp; theConfigurations.printSocketDetails ) { printSocketDetials(); } } catch (Exception e) { e.printStackTrace(); } } private static class AppConfiguration { public int port; //socket conditioning public boolean printSocketDetails; public int rcvBuf; public boolean rcvBufSpecified; public static AppConfiguration readAppConfiguration(String[] args) { AppConfiguration configurations = new AppConfiguration(); try { configurations.port = Integer.parseInt(args[0]); if ( args.length &gt; 1 ) { String[] socketConditioningDetails = args[1].split("\\|"); configurations.printSocketDetails = Integer.parseInt(socketConditioningDetails[0]) == 1 ? true : false; if ( socketConditioningDetails.length &gt; 1 ) { configurations.rcvBuf = Integer.parseInt(socketConditioningDetails[1]); configurations.rcvBufSpecified = true; } } } catch (Exception e) { System.out.println(String.format("Exception caught while parsin app configurations, %s", e.getMessage())); return null; } return configurations; } } } </code></pre> <p><strong>Client Code ...</strong></p> <pre><code>public class SimpleSocketClient implements Runnable { public static void main(String[] args) throws IOException { AppConfiguration configurations = null; if ((configurations = AppConfiguration.readAppConfiguration(args)) == null ) System.exit(1); if ( configurations.iterateThruByteRange ) { //set starting message size configurations.msgSize = configurations.startingMsgSize; for (int i = 0; i &lt; configurations.numIterations; i++) { SimpleSocketClient client = new SimpleSocketClient(configurations); client.run(); if ( configurations.msgSizeIncrementSpecified ) configurations.msgSize += configurations.msgSizeIncrement; else //double message size for next iteration configurations.msgSize *= 2; if ( configurations.reduceMsgSize ) configurations.msgCount = Math.max(1000, configurations.msgCount / 2); try { Thread.sleep(2500); } catch (InterruptedException e) { // TODO Auto-generated catch block e.printStackTrace(); } } } else { SimpleSocketClient client = new SimpleSocketClient(configurations); client.run(); } } //------------------------------------------------------------------------------ private AppConfiguration theConfigurations; private Socket theSocket; private boolean isRunning = true; private OutputStream obStream = null; private PrintWriter obWriter = null; private long[] sendTimeRecorder = null; private SimpleSocketClient(AppConfiguration configurations) { this.theConfigurations = configurations; //prepare send time recorder //sendTimeRecorder = new long[theConfigurations.msgCount]; //Arrays.fill(sendTimeRecorder, 0); } @Override public void run() { try { if ( !connectToServer() ) return; if ( theConfigurations.printSocketDetails ) printSocketDetials(); setSocketConditioning(); System.out.println(String.format("[%s] About to send msg len [%d] [%d] times ", new Date().toString(), theConfigurations.msgSize, theConfigurations.msgCount)); if ( !theConfigurations.iterateThruByteRange || ( theConfigurations.iterateThruByteRange &amp;&amp; !theConfigurations.alreadyWaited) ) { waitToContinue(); theConfigurations.alreadyWaited = true; } long startTimeInMillis = System.currentTimeMillis(); sendMessages(); System.out.println(String.format("All messages sent [%d] in timeMS [%d]", theConfigurations.msgCount, System.currentTimeMillis() - startTimeInMillis )); //printSendTimes(); } catch (Exception e) { e.printStackTrace(); } finally { try { if ( null != obWriter ) obWriter.close(); if ( null != theSocket ) theSocket.close(); } catch (Exception e) { e.printStackTrace(); } } } private void sendMessages() { //prepare fixed length message byte[] fixedLengthMessage = new byte[theConfigurations.msgSize+1]; Arrays.fill(fixedLengthMessage, (byte)0x41); fixedLengthMessage[fixedLengthMessage.length-1] = (byte)0x0D; //carriage return //long startTimeInNanos = 0; int messageCount = 0; while ( isRunning &amp;&amp; messageCount &lt; theConfigurations.msgCount ) { try { //startTimeInNanos = System.nanoTime(); obStream.write(fixedLengthMessage); //sendTimeRecorder[messageCount] = System.nanoTime() - startTimeInNanos; } catch (IOException e) { // TODO Auto-generated catch block e.printStackTrace(); } messageCount++; } } private boolean waitToContinue() { boolean exit = false; System.out.println("Hit any key to continue ..."); try { int keyValue = System.in.read(); if ( keyValue == 'q' ) exit = true; } catch (IOException e1) { // TODO Auto-generated catch block e1.printStackTrace(); } return exit; } private boolean connectToServer() { boolean connected = false; try { theSocket = new Socket(theConfigurations.host, theConfigurations.port); obStream = theSocket.getOutputStream(); obWriter = new PrintWriter(theSocket.getOutputStream(), true); connected = true; } catch (UnknownHostException e) { System.err.println(String.format("Don't know about host: %s.", theConfigurations.host)); System.exit(1); } catch (IOException e) { System.err.println(String.format("Couldn't get I/O for the connection to: %s.", theConfigurations.host)); System.exit(1); } return connected; } private void printSocketDetials() { //write to file FileOutputStream fos = null; PrintWriter pw = null; try { fos = new FileOutputStream("socket-details.log", true); pw = new PrintWriter(fos); pw.println("Socket (Client) details ...."); try { if ( null != this.theSocket ) { pw.println(String.format("IsConnected [%b]", this.theSocket.isConnected())); //SO_KEEPALIVE pw.println(String.format("getKeepAlive [%b]", this.theSocket.getKeepAlive())); //OOBINLINE pw.println(String.format("getOOBInline [%b]", this.theSocket.getOOBInline())); //SO_RCVBUF pw.println(String.format("getReceiveBufferSize [%d]", this.theSocket.getReceiveBufferSize())); //SO_REUSEADDR pw.println(String.format("getReuseAddress [%b]", this.theSocket.getReuseAddress())); //SO_SNDBUF pw.println(String.format("getSendBufferSize [%d]", this.theSocket.getSendBufferSize())); //SO_LINGER pw.println(String.format("getSoLinger [%d]", this.theSocket.getSoLinger())); //SO_TIMEOUT pw.println(String.format("getSoTimeout [%d]", this.theSocket.getSoTimeout())); //TCP_NODELAY pw.println(String.format("getTcpNoDelay [%b]", this.theSocket.getTcpNoDelay())); pw.println("----------------------------------------------------------"); } } catch (Exception e) { e.printStackTrace(); } System.out.println(); } catch (FileNotFoundException e) { // TODO Auto-generated catch block e.printStackTrace(); } finally { if ( null != pw ) { pw.flush(); pw.close(); if ( null != fos ) pw.close(); } } } private void setSocketConditioning() { try { if ( theConfigurations.rcvBufSpecified ) theSocket.setReceiveBufferSize(theConfigurations.rcvBuf); if ( theConfigurations.sndBufSpecified ) theSocket.setSendBufferSize(theConfigurations.sndBuf); if ( theConfigurations.naglingEnabledSpecified ) theSocket.setTcpNoDelay(!theConfigurations.naglingEnabled); if ( ( theConfigurations.rcvBufSpecified || theConfigurations.sndBufSpecified || theConfigurations.naglingEnabledSpecified ) &amp;&amp; theConfigurations.printSocketDetails ) { printSocketDetials(); } } catch (Exception e) { e.printStackTrace(); } } private void printSendTimes() { //write to file FileOutputStream fos = null; PrintWriter pw = null; try { String fileName = String.format("sendTimes-%d-%d-%s.log", theConfigurations.msgSize, theConfigurations.msgCount, theConfigurations.naglingEnabledSpecified &amp;&amp; !theConfigurations.naglingEnabled ? "WithNoNagle" : "WithNagle"); fos = new FileOutputStream(fileName, false); pw = new PrintWriter(fos); pw.println(Arrays.toString(this.sendTimeRecorder)); } catch (FileNotFoundException e) { // TODO Auto-generated catch block e.printStackTrace(); } finally { if ( null != pw ) { pw.flush(); pw.close(); if ( null != fos ) pw.close(); } } } private static class AppConfiguration { public String host; public int port; public int msgSize; public int msgCount; @SuppressWarnings("unused") public int msgRatePerSecond; //socket conditioning public boolean printSocketDetails; public int rcvBuf; public boolean rcvBufSpecified; public int sndBuf; public boolean sndBufSpecified; public boolean naglingEnabled; public boolean naglingEnabledSpecified; //testing byte ranges public boolean iterateThruByteRange; public int startingMsgSize; public int numIterations; public boolean reduceMsgSize; public int msgSizeIncrement; public boolean msgSizeIncrementSpecified; public boolean alreadyWaited = false; public static AppConfiguration readAppConfiguration(String[] args) { AppConfiguration configurations = new AppConfiguration(); try { configurations.host = args[0]; configurations.port = Integer.parseInt(args[1]); configurations.msgSize = Integer.parseInt(args[2]); String[] msgCountDetails = args[3].split(":"); configurations.msgCount = Integer.parseInt(msgCountDetails[0]); if ( msgCountDetails.length == 2 ) configurations.msgRatePerSecond = Integer.parseInt(msgCountDetails[1]); if ( args.length &gt; 4 ) { String[] socketConditioningDetails = args[4].split("\\|"); configurations.printSocketDetails = Integer.parseInt(socketConditioningDetails[0]) == 1 ? true : false; if ( socketConditioningDetails.length &gt; 1 ) { configurations.rcvBuf = Integer.parseInt(socketConditioningDetails[1]); configurations.rcvBufSpecified = true; } if ( socketConditioningDetails.length &gt; 2 ) { configurations.sndBuf = Integer.parseInt(socketConditioningDetails[2]); configurations.sndBufSpecified = true; } if ( socketConditioningDetails.length &gt; 3 ) { configurations.naglingEnabled = Integer.parseInt(socketConditioningDetails[3]) == 1 ? true : false; configurations.naglingEnabledSpecified = true; } } if ( args.length &gt; 5 ) { String[] byteRangeSettings = args[5].split("\\|"); configurations.iterateThruByteRange = Integer.parseInt(byteRangeSettings[0]) == 1 ? true : false; if ( byteRangeSettings.length &gt; 1 ) configurations.startingMsgSize = Integer.parseInt(byteRangeSettings[1]); if ( byteRangeSettings.length &gt; 2 ) configurations.numIterations = Integer.parseInt(byteRangeSettings[2]); if ( byteRangeSettings.length &gt; 3 ) configurations.reduceMsgSize = Integer.parseInt(byteRangeSettings[3]) == 1 ? true : false; if ( byteRangeSettings.length &gt; 4 ) { configurations.msgSizeIncrement = Integer.parseInt(byteRangeSettings[4]); configurations.msgSizeIncrementSpecified = true; } } } catch (Exception e) { System.out.println(String.format("Exception caught while parsin app configurations, %s", e.getMessage())); return null; } return configurations; } } } </code></pre> <p><strong>Command Line(s)</strong></p> <p><strong>Server:</strong></p> <pre><code>java –cp . SimpleMultiClientServer 9090 [“printSocketConditioning|rbuffer size”] </code></pre> <p>optionally <code>printSocketConditioning</code>: 1/0, prints to <code>'socketserver-details.log'</code></p> <p><code>rbuffer size</code>: receive buffer in bytes</p> <p><em>e.g. <code>java –cp . SimpleMultiClientServer 9090</code></em></p> <p><strong>Client:</strong></p> <p><strong>basic:</strong></p> <pre><code>java –cp . SimpleSocketClient localhost 9090 msgSize numMessages </code></pre> <p><code>msgSize</code>: message size in bytes <code>numMessages</code>: number of messages to send</p> <p><em>e.g. <code>java –cp . SimpleSocketClient localhost 9090 100 1000000</code></em></p> <p><strong>set socket conditioning:</strong></p> <pre><code>java –cp . SimpleSocketClient localhost 9090 &lt;msgSize&gt; &lt;numMessages&gt; “printSocketConditioning|rbuffer|sbuffer|nagle on”] </code></pre> <p><code>printSocketConditioning: 1/0</code></p> <p><code>rbuffer</code>: receive buffer size in bytes</p> <p><code>sbuffer</code>: send buffer size in bytes</p> <p><code>nagle on: 1/0</code> (equivalent to TcpNoDelay off/on)</p> <p><em>e.g. <code>java –cp . SimpleSocketClient localhost 9090 100 1000000 “1|8192|8192|1”</code></em></p> <p><strong>socket conditioning with iteration through a byte range:</strong></p> <pre><code>java –cp . SimpleSocketClient localhost 9090 msgSize numMessages [“printSocketConditioning|rbuffer|sbuffer|nagle on” [“iterateByteRange|startSize|numIterations|reduceMsgCount|incMsgSizeBy”] ] </code></pre> <p><code>iterateByteRange: 1/0</code>, perform a series of tests starting with message size at <code>startSize</code> and increment by doubling (default) or by <code>incMsgSizeBy</code>, repeat for <code>numIterations</code></p> <p><code>startSize</code>: message start size in bytes (e.g. 100)</p> <p><code>numIterations</code>: e.g. 1000000</p> <p><code>reduceMsgCount: 1/0</code>, reduce by half on each iteration or leave constant</p> <p><code>incMsgSizeBy</code>: increment message size by bytes on each iteration (e.g. 100)</p> <p>*e.g. <code>java –cp . SimpleSocketClient localhost 9090 100 1000000 “1|8192|8192|1” "1|100|20|0|100"*</code></p> <p>Note, all results post above were performed on the same hardware where both the client and server are on the same machine.</p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. USLev Levitsky
    1. USSimon
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POCan anyone explain the socket throughput discrepancies experienced between Windows (7 Pro N) and Linux ( Ubuntu 12-04)
      1. This table or related slice is empty.
      1. VTUpMod
    1. COWe need to see the benchmark code. Even though Java is cross-platform, it does so by covering over very platform specific details, so some Java classes may be implemented very differently between OSes.
      singulars
      1. POCan anyone explain the socket throughput discrepancies experienced between Windows (7 Pro N) and Linux ( Ubuntu 12-04)
      1. USIn silico
    2. COI just almost said "because it's Windows" but I saw the end of your post. :)!
      singulars
      1. POCan anyone explain the socket throughput discrepancies experienced between Windows (7 Pro N) and Linux ( Ubuntu 12-04)
      1. USLinuxios
    3. COI hope you don't have a firewall on windows ? But really, different OS have different priorities and optimizations, hard to conclude from one test, especially without code. Note that for real serious internet applications, the usual target platform is not windows.
      singulars
      1. POCan anyone explain the socket throughput discrepancies experienced between Windows (7 Pro N) and Linux ( Ubuntu 12-04)
      1. USDenys Séguret
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload