Databases
StackOverflow2013
Postsdbo
POC++ Multi threaded directory scan code
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POC++ Multi threaded directory scan code
    primarykey
    data
    text
    <p>I was looking how to write a multi threaded C++ code for scanning directory and get list of all files underneath. I have written a single threaded code which can do and below the code which can do that.</p> <pre><code>#include &lt;sys/types.h&gt; #include &lt;dirent.h&gt; #include &lt;errno.h&gt; #include &lt;vector&gt; #include &lt;string&gt; #include &lt;iostream&gt; #include &lt;sys/stat.h&gt; /* for stat() */ using namespace std; int isDir(string path) ; /*function... might want it in some class?*/ int getdir (string dir, vector&lt;string&gt; &amp;dirlist, vector&lt;string&gt; &amp;fileList) { DIR *dp; struct dirent *dirp, *dirFp ; if((dp = opendir(dir.c_str())) == NULL) { cout &lt;&lt; "Error(" &lt;&lt; errno &lt;&lt; ") opening " &lt;&lt; dir &lt;&lt; endl; return errno; } while ((dirp = readdir(dp)) != NULL) { if (strcmp (dirp-&gt;d_name, ".") != 0 &amp;&amp; strcmp(dirp-&gt;d_name, "..") != 0) { //dirlist.push_back(string(dirp-&gt;d_name)); string Tmp = dir.c_str()+ string("/") + string(dirp-&gt;d_name); if(isDir(Tmp)) { //if(isDir(string(dir.c_str() + dirp-&gt;d_name))) { dirlist.push_back(Tmp); getdir(Tmp,dirlist,fileList); } else { // cout &lt;&lt; "Files :"&lt;&lt;dirp-&gt;d_name &lt;&lt; endl; fileList.push_back(string(Tmp)); } } } closedir(dp); return 0; } int isDir(string path) { struct stat stat_buf; stat( path.c_str(), &amp;stat_buf); int is_dir = S_ISDIR( stat_buf.st_mode); // cout &lt;&lt;"isDir :Path "&lt;&lt;path.c_str()&lt;&lt;endl; return ( is_dir ? 1: 0); } int main() { string dir = string("/test1/mfs"); vector&lt;string&gt; dirlist = vector&lt;string&gt;(); vector&lt;string&gt; fileList = vector&lt;string&gt;(); getdir(dir,dirlist,fileList); #if 0 for (unsigned int i = 0;i &lt; dirlist.size();i++) { cout &lt;&lt; "Dir LIst" &lt;&lt;dirlist[i] &lt;&lt; endl; //string dirF = dir + "/" + dirlist[i]; //getdir(dirF,fileList); } #endif for (unsigned int i = 0; i &lt; fileList.size(); i++) cout &lt;&lt; "Files :"&lt;&lt;fileList[i]&lt;&lt; endl; return 0; } </code></pre> <p>Now issue is that it is single threaded and I need to scan say about 8000 directories under which file can be present. So I am not getting how to do so as number of directories can vary as it is decided by N dimension matrix.</p> <p>Any help in this regard will be great. Thanks in advance.</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USchrisaycock
    1. USAbhinav
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POC++ Multi threaded directory scan code
      1. This table or related slice is empty.
      1. VTUpMod
    1. COI'm not quite sure what the N-dimension matrix is that you refer to. But to answer your question, you ultimately have to do a parallel depth-first search. I assume you know how to do multithreaded programming already; if not, then you need to learn about that, preferably with the Boost library rather than just POSIX threads.
      singulars
      1. POC++ Multi threaded directory scan code
      1. USchrisaycock
    2. COBasically with N-dimension matrix I want to say let say we have a 20x20x20 matrix then I will have 8000 directories which need to be searched. So this can be increased based on requirements.
      singulars
      1. POC++ Multi threaded directory scan code
      1. USAbhinav
    3. COTo get good performance, whatever you define that to be for this application, ultimately your hardware (and OS) will also have to support this. Usually I/O is a bottleneck for multithreaded apps, even if you split up the work into different threads, they tend to all wait for access to the same I/O, thus limiting any parallelism you hope to achieve.
      singulars
      1. POC++ Multi threaded directory scan code
      1. USChris O
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload