Data mining one's own library

[Jasso]

Here's an issue I hope other Bookends users might be able to help me with. Not really a problem, but more of a question of optimising work.

After collecting references into Bookends for several years, I have a pretty good reference library sitting on my laptop. And not just references – out of about 1900 refs, more than 1400 have the actual articles (or even full books I've scanned) attached in them.

So I've come to realise that whenever I start thinking about any particular topic, the first thing I should do is to check my own library. I might just have a good basic collection already which could then be easily supplemented with searching for recent articles.

But. The problem is that not all of the references have proper keywords in them. This means that inside Bookends my search abilities are somewhat limited.

A recent example: I was writing a paper and wanted a make a quick remark on the so called Standard Social Science Model (SSSM). But searching for the concept or its abbreviation inside Bookends comes up empty as it is not in the keywords or in any of the headlines or abstracts. But I know it appears in several papers of which I have full pdfs in my library.

So this got me thinking, once again, about how to data mine my own library for information. Sure, I can do a Spotlight search. But the problem there is that it gives me too many results, most of which are actually not that useful because they might be printouts from web pages etc.So I end up wasting time opening useless articles and possibly don't even have the energy to check all of the results (thus missing out some good ones).

What I want is just results from my Bookends library, which I know to be of good quality articles which I can actually use. Then I know it's pretty much worth going through every article my search comes up with.

And because Bookends does not have a "Search within library pdf content" option (or does it?), I'm thinking there must a way to create that somehow. With Apple script or something? And what would be the best way to incorporate it into your workflow?

Yes, I do realise one could do a Spotlight search within that folder, but I'm thinking about creating some kind of shortcut to that. And perhaps something which has more search options. Just like when using a proper library database.

Any ideas?

[nicka]

Isn't the Spotlight search in Bookends exactly what you are asking for? See pp 89-90 of the user guide.

[Jasso]

For whatever reason, choosing the "Spotlight" from the search menu and trying to search doesn't return the results I know are there.

Also, I don't want to scroll down the list and choose the separate Spotlight search every time I want to find something in the library. That's what I meant by saying that this is more about optimising workflow than something which couldn't be done at all.

I guess for me, the optimal solution would be to have two separate search fields in the toolbar. And as this is not possible, I'm trying to come up with similar good ways to do this.

[nicka]

About the missing results: I remember hearing that Spotlight only indexes the first n pages of long documents (for some value of n that I don't remember). Does anyone know if that is (still) true?

[Jon]

Use Find (Command-F). You can combine a Spotlight search with a text search of the database (even save as a smart group if you want).

And don't forget using the tag clouds to find references. Bookends will generate tag clouds from titles, abstracts, and notes if you want.

As for finding the pdfs:

1. They myst be in the default attachment folder (or subfolder).
2. They must have "readable" text (if you scanned without OCR, they can't be indexed).
3. Since the searches are created "robotically", not all will find a match. Spotlight can be finicky about what it will consider a match and what it won't.

Jon
Sonny Software

[Jasso]

Thanks for the tip! ... But I tried this in Bookends and for some reason it didn't work.

Here's what Spotlight search of the Attachments folder gives me right after running the search. Of course, more results might pop up if I wait. These delays and the finickiness you mentioned makes me dislike Spotlight alone for this task.

search01.png (10.2 KiB) Viewed 9465 times

When I copy-paste the same search terms in Bookends Cmd-F and run it, it turns out blank with nothing in the hits list. (Not including the screenshot of the blank Hits list, but yes, I did click Find! ) Am I doing something wrong here?

search02.png (15.76 KiB) Viewed 9465 times

[Jon]

Try removing the quote marks. If still no luck, and the hits that you found in the Finder are in the Bookends default attachment folder, send me one of those pdfs (support@sonnysoftware.com).

Jon
Sonny Software

[Jon]

I see that using quote marks will cause Bookends' Spotlight searches to return nothing (this is not new behavior, it's in all versions of Bookends). I've fixed this for the next Bookends update (after 12.0.1).

Jon
Sonny Software

[Jasso]

Great! Looking forward to this.

For a temporary fix I made an Automator workflow which allows me to perform Spotlight searches inside Bookends attachments folder with one click.

[Jon]

It just posted a silent update (evening of July 5) with this, and a few formatting fixes.

Jon
Sonny Software

[Jasso]

That was fast, thank you!

Now it works perfectly, and launching the search with Cmd-F fits perfectly to my definition of a quick and simple workflow.

Ps. Thanks for also fixing the duplication of page numbers in citations in the same update. My custom formats are working again.