Databases
StackOverflow2013
Postsdbo
PO
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <p>The hard part of this is the scanning of the directory, just because it can be expensive.</p> <p>But that's a cruel reality since you can't use inotify et al.</p> <p>In your database, simply create a node type record:</p> <pre><code>create table node ( nodeKey integer not null primary key, parentNode integer references node(nodeKey), // allow null for the root, or have root point to itself, whatever fullPathName varchar(2048), nodeName varchar(2048), nodeType varchar(1) // d = directory, f = file, or whatever else you want ) </code></pre> <p>That's your node structure.</p> <p>You can use the full path column to quickly find anything by the absolute path.</p> <p>When a file moves, simply recalculate the path.</p> <p>Finally, scan you music files. In unix, you can do something like:</p> <p>find . -type f | sort > sortedListOfFiles</p> <p>Next, simply suck all of the path names out of the database.</p> <p>select fullPathName from node where nodeType != 'd' order by fullPathName</p> <p>Now you have two sorted list of files.</p> <p>Run them through DIFF (or comm), and you'll have a list of deleted and new files. You won't have a list of "moved" files. If you want to do some heuristic where you compare new and old files and they have the same endings (i.e. ..../album/song) to try and detect "moves" vs new and old, then fine, no big deal. Worth a shot.</p> <p>But diff will give you your differential in a heartbeat.</p> <p>If you have zillions of files, then, sorry, this it going to take some time -- but you already know that when you lose the inotify capability. If you had that it would just be incremental maintenance.</p> <p>When a file moves, it's trivial to find its new absolute path, because you can ask its parent for its path and simply append your name to it. After that, you're not crawling a tree or anything, unless you want to. Works both ways.</p> <p>Addenda:</p> <p>If you want to track actual name changes, you can get a little more information.</p> <p>You can do this:</p> <pre><code>find . -type f -print0 | xargs -0 ls -i | sort -n &gt; sortedListOfFileWithInode </code></pre> <p>The -print0 and -0 are used to work with files with spaces in them. Quotes in the file names will wreck this however. You might be better off running the raw list through python and fstat to get the inode. Different things you can do here.</p> <p>What this does is rather than just having names, you also get the inode of the file. The inode is the "real" file, a directory links names to inodes. This is how you can have multiple names (hard links) in a unix file system to a single file, all of the names point to the same inode.</p> <p>When a file is renamed, the inode will remain the same. In unix, there's a single command used for renaming, and moving files, mv. When mv renames or moves the file, the inode stays the same AS LONG AS THE FILE IS ON THE SAME FILE SYSTEM.</p> <p>So, using the inode as well as the file name will let you capture some more interesting information, like file moves.</p> <p>It won't help if they delete the file and add a new file. But you WILL (likely) be able to tell that it happened, since it is unlikely that an old inode will be reused for the new inode.</p> <p>So if you have a list of files (sorted by file name):</p> <pre><code>1234 song1.mp3 1235 song2.mp3 1236 song3.mp3 </code></pre> <p>and someone removes and adds back song 2, you'll have something like</p> <pre><code>1234 song1.mp3 1237 song2.mp3 1236 song3.mp3 </code></pre> <p>But if you do this:</p> <pre><code>mv song1.mp3 song4.mp3 </code></pre> <p>You'll get:</p> <pre><code>1237 song2.mp3 1236 song3.mp3 1234 song4.mp3 </code></pre> <p>The other caveat is that if you lose the drive and restore it from backup, likely all of the inodes will change, forcing effectively a rebuild of your index.</p> <p>If you're real adventurous you can try playing with extended file system attributes and assign other interesting meta data to files. Haven't done much with that, but it's got possibilities as well, and there are likely unseen dangers, but...</p>
    singulars
    1. This table or related slice is empty.
    1. POI want a clever algorithm for indexing a file directory...pointers?
      singulars
      1. PTQuestion
    1. PTAnswer
    1. USWill Hartung
    1. USWill Hartung
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. POI want a clever algorithm for indexing a file directory...pointers?
      singulars
      1. PTQuestion
    1. This table or related slice is empty.
    1. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTAcceptedByOriginator
    1. COthanks! the unix `find` command is lightning fast - much better than python. what do you mean by DIFF? I searched and couldn't find it in the MySQL docs
      singulars
      1. PO
      1. USlollercoaster
    2. COoh nvm. you meant the unix diff command (i think). the only problem this doesn't solve is that if files are renamed, the playlists table in the database won't be able to keep track of which paths belong to which playlist - I won't have a way to track them. Unless I'm mistaken.
      singulars
      1. PO
      1. USlollercoaster
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload