Databases
StackOverflow2013
Postsdbo
POWhy is my code using monadic lists from the List package so slow?
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POWhy is my code using monadic lists from the List package so slow?
    primarykey
    data
    text
    <p>Last week user Masse asked a <a href="https://stackoverflow.com/questions/3881835/how-to-list-directories-faster">question about recursively listing files</a> in a directory in Haskell. My first thought was to try using monadic lists from the <a href="http://hackage.haskell.org/package/List" rel="nofollow noreferrer"><code>List</code> package</a> to avoid building the entire list in memory before the printing can start. I implemented this as follows:</p> <pre><code>module Main where import Prelude hiding (filter) import Control.Applicative ((&lt;$&gt;)) import Control.Monad (join) import Control.Monad.IO.Class (liftIO) import Control.Monad.ListT (ListT) import Data.List.Class (cons, execute, filter, fromList, mapL) import System (getArgs) import System.Directory (getDirectoryContents, doesDirectoryExist) import System.FilePath ((&lt;/&gt;)) main = execute . mapL putStrLn . listFiles =&lt;&lt; head &lt;$&gt; getArgs listFiles :: FilePath -&gt; ListT IO FilePath listFiles path = liftIO (doesDirectoryExist path) &gt;&gt;= listIfDir where valid "." = False valid ".." = False valid _ = True listIfDir False = return path listIfDir True = cons path $ join $ listFiles &lt;$&gt; (path &lt;/&gt;) &lt;$&gt; (filter valid =&lt;&lt; fromList &lt;$&gt; liftIO (getDirectoryContents path)) </code></pre> <p>This works beautifully in that it starts printing immediately and uses very little memory. Unfortunately it's also dozens of times slower than a comparable <code>FilePath -&gt; IO [FilePath]</code> version.</p> <p>What am I doing wrong? I've never used the <code>List</code> package's <code>ListT</code> outside of toy examples like this, so I don't know what kind of performance to expect, but 30 seconds (vs. a fraction of a second) to process a directory with ~40,000 files seems much too slow.</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USCommunity
    1. USTravis Brown
    plurals
    1. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POWhy is my code using monadic lists from the List package so slow?
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POWhy is my code using monadic lists from the List package so slow?
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POWhy is my code using monadic lists from the List package so slow?
      1. USDavorak
      1. VTFavorite
    1. COWe could use a `Data.ByteString.getDirectoryContents :: ByteString -> [ByteString]`. When you consider 40,000 files at 10+ characters each, that's 400,000 characters. A [Char] in Haskell takes what? Ballpark 12 bytes (x86) or 24 (x86-64). So those 400,000 characters are 8MB or more spread throughout a linked list. Now that I've said all this, hopefully you will respond with "I've benchmarked getDirectoryContents and that isn't the issue".
      singulars
      1. POWhy is my code using monadic lists from the List package so slow?
      1. USThomas M. DuBuisson
    2. CO@TomMD: I'm primarily interested in the comparison with `FilePath -> IO [FilePath]` versions that use `getDirectoryContents` in exactly the same way (Masse's original implementation, for example). So I don't think this is the problem, but I'll take a look.
      singulars
      1. POWhy is my code using monadic lists from the List package so slow?
      1. USTravis Brown
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload