Speed issues on attachment with large folder hierarchy

[slandrum]

Hi---

I'm using 10.4.1, and I really like the recently-added feature of showing the full folder hierarchy of the attachments folder before picking a folder to file the current attachment in. However, I have a *really large* folder hierarchy where I file Bookends PDFs, and I'm experiencing a long lag between clicking "attach" and the display of the attach dialog, and then another long lag after I select the folder to file this PDF in. (On the order of 10-15 seconds, which really cramps my ability to work quickly through sources for my dissertation.)

I'd love to see the ability to customize how deep Bookends follows the folder hierarchy in the attachments folder, or some intelligent caching of that directory hierarchy so that the lag won't be so long.

[Jon]

Hi,

If Bookends has to iteratively search through thousands of files/folders, it can slow things down (how much depends on factors like which OS you are using and how fast your Mac is). You do control the level of hierarchy -- if you limit yourself to creating just subfolders, for example, Bookends doesn't have to look through sub-subfolders.

As for caching -- it might be possible to do that, but if you were to add a subfolder in the Finder (not via Bookends) the cache wouldn't be updated.

Jon
Sonny Software

[slandrum]

Good point about the reasons for not caching--- at least not without a good way to decide when to regenerate the cache.

The sub-subfolders thing is complicated. You're right, I do control the structure---- but not all of my sources are easily handled by Bookends. I work with lots of archival collections--- a type that's notoriously hard to cite programmatically, particularly in Chicago Humanities, so I use the filesystem to replicate the collections' physical structure (for filing PDFed photocopies and digicam photos of items in the collections.) A given archival repository can have multiple collections, and each collection can have several boxes, and each box can have several folders, and each folder can have several items. One item can have several images, if it's a multipage document; I could PDF them together into one document, but I prefer handling JPGs in folders.

When I want to cite a particular item from an archival collection, I create a Bookends record, but I don't move the files when I attach them; I just make a subfolder for each item (within its repository, collection, box, and folder) and name it appropriately. Those deep sub-sub-subfolders serve an organizational purpose for me; they preserve citation information that I don't have time to automate, for thousands of items.

Currently, my "sources" folder contains both those deep hierarchies for archival sources as well as my Bookends-cited articles. It's a quick fix for me to move the "Archival" folder out of "sources," but it disrupts the human-readable organizational value of filing everything together, and it makes handling Archival images with Bookends much more difficult. I recognize that my situation's probably an edge case for your user pool, but it'd be nice to not have to work around a feature that's otherwise very useful.

(BTW, I've got a reasonably fast machine, a 1.2Ghz Intel Macbook, so a 10-second delay should give you some idea of the size of the folder hierarchy I'm dealing with.)

[Jon]

Thanks for the explanation.

I don't think limiting the subfolder search depth is a good approach. First, it's a solution that is highly specific for your particular filing scheme. And it's going to break if you ever do have some sub-subfolders that actually contain pdfs.

Folder path caching would probably work satisfactorily, with that caveat that the cache/folder list wouldn't be updated if you add to the subfolder hierarchy in the Finder. It might be possible to have a "Rebuild Folder List" in the Attachment dialog subfolder popup menu, which would let you deal with this manually...

Jon
Sonny Software

[slandrum]

Folder path caching would probably work satisfactorily, with that caveat that the cache/folder list wouldn't be updated if you add to the subfolder hierarchy in the Finder. It might be possible to have a "Rebuild Folder List" in the Attachment dialog subfolder popup menu, which would let you deal with this manually...

That sounds like a good solution. It'd also be nice to have a UI option that works more smoothly than a pulldown in cases with deep hierarchies; a pulldown with 250 items is a bit inelegant when trying to scroll down to item 249. Any possibility of using a tree-style navigation panel?

[Jon]

I hadn't thought of that. Can you tell me what you had in mind (perhaps point to an application)?

Perhaps you can contact me directly for this, and for testing the implementation (support@sonnysoftware.com).

Jon
Sonny Software

[whshep]

I would second the vote for some sort of tree-style navigation in the attachment folder view, or anything that might make navigating the hierarchy easier. Scrolling down a long list can be difficult. One possibility might be keyboard navigation, a la the Finder (i.e., pressing K takes you to the folders beginning with K).

To explain why this might be useful: I have a fairly large collection of PDFs, dating to prior to my use of Bookends. I also use DevonThink, and index these PDFs in DT (rather than letting DT import and manage them--that's another story). My PDF file system is arranged by source (journal, website, book title, etc.). I use a folder action to index in DT anything added to a folder automatically. So my usual system of working is: create reference in BE; create folder for PDF (if needed); manually put PDF in folder (which automatically indexes it in DT); manually attach PDF to BE reference.

BE's new ability to navigate my file system will save me some time here: I can automatically import the PDFs into BE and the file system (and thus DT) at the same time—all I have to do is create the necessary folder in the Finder beforehand, if needed. BE will also automatically change the filename for me as well, which is a real nice feature for those PDFs that download with a long string of numbers rather than a decipherable name.

In short, this is a great new feature and a boon to BE users (great job, Jon!). But more ease of navigation would make it even more useful.

And yes, I've thought of the disadvantage of the keyboard navigation idea: Right now, pressing R takes you to Rebuild Folder List, and we still need to get to that quickly.

[Jon]

Hi,

The problem is that keyboard navigation of popup menus is handled by the OS, not Bookends. Bookends gives an indented list so you can see the folder hierarchy, and to do this spaces are used to create the offset. You can have one or the other (a structured list or a keyboard-navigable one), but not both. I think there would have to be a completely different approach to managing the attachment subfolder list.

Jon
Sonny Software

[gnoli]

whshp,
if you want, you can read this post by myself on DT forum (I have a situation similar to your, I think)

http://www.devon-technologies.com/scrip ... f=2&t=7392

[whshep]

if you want, you can read this post by myself on DT forum (I have a situation similar to your, I think)

http://www.devon-technologies.com/scrip ... f=2&t=7392

See my reply to your post, there's a script there that will probably help.

[whshep]

You can have one or the other (a structured list or a keyboard-navigable one), but not both. I think there would have to be a completely different approach to managing the attachment subfolder list.

How about a good-old-fashioned "Open" dialog box? That would not only allow us archivist types to access the various levels of the hierarchy, but also to use the Finder to create new subfolders, and then drag them into the Open box.

At any rate, the new system has greatly simplified my workflow, thanks so much!

[Jon]

Hm, that's an interesting idea. Perhaps a browse button next to the popup menu for power users? Bookends would have to reject selections that were outside of the top-level folder, but that wouldn't be a big deal. Let me think about this. Thanks for the suggestion.

Jon
Sonny Software

[whshep]

Let me think about this. Thanks for the suggestion.

I love it—not only does the suggestion get a thought, it gets implemented in the next revision.

When people ask me, Why Bookends? this is the kind of thing I tell them.