Sonny Software

There's a lot I love about Bookends... once references are in Bookends. Unfortunately, I continue to pine for Zotero when I'm searching for and adding references to my library. I'm hoping I'm missing something about this process.

I do most of my search/discovery/browsing for references in my browser. (Opening up the Bookends browser just for research is counterintuitive to me. From a browser, Zotero's import is blazingly fast and incredibly convenient. Once I've found a reference I want, I click the Import to Zotero Safari Extension and wait a maximum of ten seconds. Then the reference is added and a snapshot or PDF is downloaded, no matter what kind of reference I'm pulling (as it works on news articles, journal articles, book listings… you name it). In the worst case, it at least autofills the item's title and URL. Similarly, if I drag a PDF into Zotero, it finds the reference info and autopopulates metadata within ten seconds while I watch.

Here's a demo video of Zotero on three different reference types. This has worked just as pictured ~95% of the time on thousands of references over the years.



(sorry for the terrible quality, and for the annoying circling of the mouse. I was trying to draw attention to the fact that all the metadata is populated within seconds...)

Bookends is a different story. Once I've found a reference in Safari, I tap the "Open in Bookends" bookmarklet. Since the last version or two of Safari, I have to click "Allow". Then I have to wait for Bookends Browser to open with the URL. Once it's loaded (for a second time, since I already had it loaded in Safari!) I right-click on the Attachment proxy and select "Attach web page to new empty reference in library and autocomplete". Then, most of the time, Bookends won't be able to autocomplete right away and a search window will appear. Then I have to re-type or copy+paste the item title and search. Then I have to select the right item and hit Import. And that will only work if the reference is a scholarly work. If it's on a newspaper or blog, it will fail to pull anything (not even a title!) and I will have to manually add those details.

The contrast between these two experiences is so awful that I assume I'm doing something incredibly wrong. I also haven't seen anyone else talk about the differences, so I'm convinced of my own incompetence. I've tried to figure out what I should be doing, but have failed even though I've been using Bookends for over a year.

So, please, someone, tell me that there's a better way to import... or Jon, is there any hope we'll ever get a browser extension or an upgrade to the automaticity of this process?

Otherwise, maybe I can divide the workflow, using Zotero to collect references and then script the import to Bookends once every few hours or something...

There are few things I'd do differently. First, start your search in Bookends Browser, not Safari. That would save the initial steps. Second, don't right-click the proxy PDF, click on the downward arrow that appears when Bookends identifies reference metadata in the web page. That will list the references it finds for you to download, with an option to try to obtain the PDF as well. Only if the downward arrow doesn't appear (which means Bookends doesn't fine a DOI or JSTOR URL) would I go the route you're using now.

Jon
Sonny Software

Right, of course, the downwards arrow. Sorry. I should have described the ideal workflow in Bookends.

Trouble is, it doesn't appear very often for me. Half the time would be an optimistic guess.

Zotero's magic works 95% of the time, even on non-scholarly sources (which I use often). So I guess, is there any hope of seeing some development in this area?

Zotero's open source. I tried looking around a little to figure out how it works to no avail, but maybe I'll try some more. Perhaps its import process can be scripted in Bookends..?

AFAICT Zotero, which is a relatively large effort backed by a university and with many contributors, uses a variety of methods including HTML scraping to obtain metadata. We can't do that for a lot of reasons. Bookends uses unique identifiers in the web page (DOIs, JSTOR URLs) or embedded COinS to get the metadata automatically. If none are found Autocomplete paper is the fallback. We also recently updated Bookends Browser to work with Google Scholar so that any references you find there can be directly imported into Bookends (with PDFs, if available). We're always thinking of ways to improve retrieval of reference metadata, but using brute force isn't something we can do.

Jon
Sonny Software

HTML scraping is what I was guessing, too. At least on the non-scholarly source side. They must also be doing some kind of fuzzy content matching for scholarly PDFs. Zotero often succeeds when Bookends can't autocomplete, but once in a while Zotero fails spectacularly and downloads metadata for the completely wrong paper.

Thanks for the responses. Count me in for a few feature requests, then:
- A more robust Bookends Browser (e.g., with tabs and windows);
- A Bookends browser extension in Safari with better integration (e.g., no hitting "allow", no reloading the page in Bookends Browser); and
- Some filling in of data when autocomplete doesn't find a DOI, JSTOR, or COinS match (even putting the HTML page title in the Title box of the search window/metadata info would save many keystrokes and clicks!)

Thanks for the quick responses. I'll try to find other ways of speeding up this process.

@ryanjamurphy

This won't substitute the speedy workflow of Zotero (I've moved from it to Bookends several year ago and also miss its fast capture abilities), but maybe it can useful.

I gave up on Bookends browser because it felt too slow for my needs. What I do now is:

Step 1. Use safari to research and download pdf files (for all the other formats I use DEVONthink to capture and store; with a few exceptions to some epubs).

Step 2. I download the pdf to a specific folder (that I call Import to Bookends), which is set as (File> Import From) Watch Folder in Bookends. This automatically imports the file to Bookends.

Note: I try to direct download the pdf to this folder using a Default Folder X shortcut. Nonetheless, because some repositories download the file automatically without your 'save as' input, you can either (a) set a rule in Hazel to send all pdf files with doi.org content to the Watch Folder, or/and (b) keep the link to the Watch Folder in Finder sidebar for fast drag and move).



Step 3. The pdf is then imported to Bookends. If the pdf is a new one, the embedded metadata is automatically inserted in the reference (you just have to complete what is missing). If it doesn't complete anything, you can select the (incomplete) reference and hit (Refs>)Autocomplete Paper. Fill up the basic data (e.g. title) into the searching database (e.g. Google Books or Google Scholar) and most of the reference should be completed.



Step 4. You can also set the autocomplete to include renaming the file or not (I normally prefer to verify if all is okay and then do the renaming by shortcut (Refs>Global Changing... I set a shortcut for this via Mac System Preferences > Keyboard).

Step 5. To speed things a bit more, I also make full usage of the Mac toolbar (with BetterTouchTool). In it I have a few touch shortcuts (see image below): the orange one are to copy citation, formatted and hyperlink, and a merged version of formatted and hyperlink (that I set with Keyboard Maestro and I use it to connect the Bookends reference to my DEVONthink Zettelkasten's database); the red ones are for quick add reference, rename and autocomplete. Another way is to set a Keyboard Palette with these more common actions.



The system is not perfect, but I now find it super fluid and relatively fast. Maybe you can adapt something like this for you.

Thanks, @mjvalente, this is insightful.

Believe it or not, I've been using both Zotero and Bookends these days. My approach is...
Search phase
While searching for materials for a given need:
- Find an article
- Use the Save to Zotero Safari Extension to clip any scholarly reference into Zotero
- Repeat

Then, once I'm finished a "search" phase:
Export to Bookends
In Zotero:
- Select all newly added items in Zotero
- Context menu → Export Items
- RIS Format, save Notes and Files checked
- Save to Desktop
In Bookends:
- File → Import References → From File or Clipboard...
- Select the .RIS file exported from Zotero
In Zotero:
- Delete the references.

That's it. I do this for anywhere from one reference to dozens. It may not be faster for one PDF in which the metadata is readily available in Bookends, but in the general case, it does seem to be an easier and faster workflow. As a bonus it reduces cognitive load, as I'm only ever doing one procedure to save scholarly references.

This is sad, I realize, as it's wonky. Again, I hope Bookends can get faster on the import front in future releases. Zotero is positively blazing in contrast!

This seems to depend on your field. At least for the biomediacal sciences, almost all potential references will have a DOI or pubmed ID, and in this case there is no advantage to Zotero AFAICT? Personally I search in my browser, then simply select the PMID or DOI and quick add (in my case triggered via Alfred). Bookends Quick add uses pubmed and crossref as a backup and will find anything that has been registered. Apart from auto-selecting the text, what else can be optimised?

I could see it being field-dependent. I am fairly transdisciplinary, with information systems/management science being the most popular arena of my work. Not only do papers sometimes fail in Bookends's autocomplete, but I also cite non-scholarly references often.

I will check out the Alfred workflow, though, thanks!