[Feature Request] Auto-sync to a BibTeX File

A place for users to ask each other questions, make suggestions, and discuss Bookends.
iandol
Posts: 465
Joined: Fri Jan 25, 2008 2:31 pm

[Feature Request] Auto-sync to a BibTeX File

Post by iandol »

Those of us who either write in LaTeX or use Markdown tools like Pandoc have to maintain a .bib "copy" of our Bookends database. It is possible to automate exporting using applescript (though there is a ongoing memory leak (in WindowServer, not Bookends) using applescript for export for large libraries, and it is slow). Otherwise we can do this manually every week or so, and though this is not a major issue, it would be nice if this was automatic. On testing the Mendeley desktop app (importing ~6900 refs from Bookends), I noticed it offers an automatic sync feature:
Screen Shot 2020-09-20 at 18.37.27_SML.png
Screen Shot 2020-09-20 at 18.37.27_SML.png (33.03 KiB) Viewed 27832 times
On my brief testing their feature seems to work well. I imagine us BibTeX file users are in the minority of Bookends users, and developing a sync to file (so only new refs are added) is a non-trivial feature to add, but it would nevertheless be nice to have in Bookends.
Jon
Site Admin
Posts: 10048
Joined: Tue Jul 13, 2004 6:27 pm
Location: Bethesda, MD
Contact:

Re: [Feature Request] Auto-sync to a BibTeX File

Post by Jon »

This has been discussed on the forum (several times, I think). It's not possible because, as you say, it's slow. Imagine Bookends formatted 1000 references as BibTeX every time you change a comma in a field.

I'm guessing (really just a guess) that Mendeley keeps a BibTeX-formatted copy of each reference in the database and writes them to a file with each reference import/edit (which would be much faster, because the formatting has already been already done). That adds lots of overhead to benefit a small number of users. Since Bookends can export the necessary file on demand, with or without an AppleScript, it's unlikely we would implement such a feature.

Jon
Sonny Software
iandol
Posts: 465
Joined: Fri Jan 25, 2008 2:31 pm

Re: [Feature Request] Auto-sync to a BibTeX File

Post by iandol »

Indeed, it seems this is the most comprehensive previous thread also inspired by Mendeley's feature from 2016: viewtopic.php?p=18492#p18492 —— which I even posted on and completely forgot about. :oops:

Theoretically, this doesn't have to be slow —— Bookends does something similar with its iCloud support. Synchronisation is atomic, only refs that change get refreshed in the background. This is increasingly common for apps that rely on the cloud, running sync parallel to other operations and with whole programming paradigms dedicated to this kind of work (async coroutines working separately from a main thread etc.)

However, for example the iCloud sync mechanism has been designed and coded by Apple (probably 10s-100s of dedicated programmers), and I do realise you'd need to write a sync mechanism from scratch which is a technical challenge, and writing to a local structured text file is different to using a database for data storage. Thanks for your consideration anyway!
Nhaps
Posts: 244
Joined: Mon Sep 26, 2011 10:05 pm

Re: [Feature Request] Auto-sync to a BibTeX File

Post by Nhaps »

Bookends is a swiss knife that conforms to many text editors but does not fit nicely to the needs of LaTeX users. For that I'm using JabRef, which offers native bib sync right out of the box. Also, the creation and maintenance of reference fields is more flexible, in my view. I actually use both reference managers, for different needs.
iandol
Posts: 465
Joined: Fri Jan 25, 2008 2:31 pm

Re: [Feature Request] Auto-sync to a BibTeX File

Post by iandol »

Jabref has some really nice management features, I agree! I actually utilise Jabref to cleanup my Bookends library, the "Check Integrity" tool returns a brilliant "health report" on the state of the database items like duplicate DOIs, malformed journal names, rogue HTML entities and lots of other checks. I thus export .bib from Bookends, load it into Jabref to run the report then find refs in Bookends to clean up. It works super fast directly on the .bib file even with thousands of references. The options to manage journal abbreviations are brilliant, as is the interface to merge online information into an existing reference. Really nicely designed and powerful. But, the workflow with bookends overall is better, management of author names is better, global changes are easier, note taking from PDFs more refined, the iOS app is brilliant, and Bookends is much easier to integrate into my workflow, so that the small inconvenience of having to regenerate the bib file is worth it... But yes, Jabref is an excellent choice especially for LaTeX / Pandoc users otherwise.
Nhaps
Posts: 244
Joined: Mon Sep 26, 2011 10:05 pm

Re: [Feature Request] Auto-sync to a BibTeX File

Post by Nhaps »

Jon wrote: Sun Sep 20, 2020 7:43 am This has been discussed on the forum (several times, I think). It's not possible because, as you say, it's slow. Imagine Bookends formatted 1000 references as BibTeX every time you change a comma in a field.

I'm guessing (really just a guess) that Mendeley keeps a BibTeX-formatted copy of each reference in the database and writes them to a file with each reference import/edit (which would be much faster, because the formatting has already been already done). That adds lots of overhead to benefit a small number of users. Since Bookends can export the necessary file on demand, with or without an AppleScript, it's unlikely we would implement such a feature.

Jon
Sonny Software
I suppose the reason for the small number of users is precisely because BE does not offer auto-sync. A lot of people in Academia use LaTeX and/or Markdown, and could potentially buy BE if this feature could be implemented somehow. What ideas come to mind? My suggestion is that the whole library of references does not need auto-sync necessarily, but a static group one is working with. Similar to what has been implemented in Watch Folder.
DrJJWMac
Posts: 345
Joined: Sat Jun 22, 2019 8:04 am
Location: Alabama USA

Re: [Feature Request] Auto-sync to a BibTeX File

Post by DrJJWMac »

At a minimum, I would appreciate any efforts to help avoid that we need to go to outside tools (e.g. JabRef) to do additional checking on the integrity of a Bookends database (referencing the comment from @landol last year).

To counter @Nhaps, as one who uses BibTeX, I am not immediately interested in whether Bookends includes an auto-sync option (I think the discussion here BTW is about auto-export, not auto-sync, since the BibTeX file is not "re-imported" back to Bookends if it changes). I find the manual option to export to BibTeX approach works well enough.

Finally, I believe the argument that more folks in academia would purchase Bookends if it included an auto-sync option may be stretching the limit of reasoning on why to implement the feature. Folks in academia (as I am) can tend to be rather particular about their favorite software tools. They will choose one or the other based on considerations that do not always include in-built automation, especially when the manual options that could be automated already are flexible and efficient. In this regard, let's also face it that the only productive macOS options for citation management are Mendeley, ReadCube Papers, Bookends, JabRef, and Zotero and that these tools have distinctively different approaches that go well beyond whether the app has automatic sync with a BibTeX file. The analogy here is saying the more families with children will buy a base-level Ford SUV rather than a base-level Honda SUV when the former would include an automatic, daylight/night-time sensing rear-view mirror rather than a manual one.
--
JJW
msteffens
Posts: 37
Joined: Thu Jul 19, 2007 10:04 am
Location: Germany
Contact:

Re: [Feature Request] Auto-sync to a BibTeX File

Post by msteffens »

iandol wrote: only refs that change get refreshed in the background.
Years ago, I've implemented a sync with a BibTeX file for refbase where found records get appended to a local BibTeX file (if they don't yet exist in that file), and existing records in that file get updated if their modification date on the server is more recent.

A similar mechanism could be used with a script for Bookends. I.e., get all publication items that were last modified since XX hours/days/etc, and export only these to a list of BibTeX records. This could be done like this:

Code: Select all

property lastSyncDate : missing value
property hoursToLookBack : 6

if lastSyncDate is missing value then
	set curDate to current date
	set lastSyncDate to curDate - (hoursToLookBack * hours)
end if

tell application "Bookends"
	tell front library window
		set lastEditedPubs to every publication item whose date modified is greater than lastSyncDate
		
		if lastEditedPubs is not {} then
			set bibtexContent to format lastEditedPubs using "BibTeX.fmt" as BibTeX
			set the clipboard to bibtexContent -- DEBUG (only for display purposes)
		end if
	end tell
end tell
Then, let the script walk each BibTeX record, and add a note like this:

Code: Select all

optnote="exported from Bookends (publication item id 493), last updated on Wed, 29 Sep 2021 19:15:35 +0100",
Finally, have a regular expression pattern search & replace matching BibTeX records in the existing BibTeX file, and append any BibTeX records that didn't match. This last processing step may be a bit nasty, but it's doable and can be made accurate.

Using third-party tools, the script could be made to execute at recurring time intervals, or it could simply be called manually via a keyboard shortcut.

I could look into writing such a script. As always, the biggest problem is mostly time available for development…
Nhaps
Posts: 244
Joined: Mon Sep 26, 2011 10:05 pm

Re: [Feature Request] Auto-sync to a BibTeX File

Post by Nhaps »

DrJJWMac wrote: Wed Sep 29, 2021 9:27 am At a minimum, I would appreciate any efforts to help avoid that we need to go to outside tools (e.g. JabRef) to do additional checking on the integrity of a Bookends database (referencing the comment from @landol last year).

To counter @Nhaps, as one who uses BibTeX, I am not immediately interested in whether Bookends includes an auto-sync option (I think the discussion here BTW is about auto-export, not auto-sync, since the BibTeX file is not "re-imported" back to Bookends if it changes). I find the manual option to export to BibTeX approach works well enough.

Finally, I believe the argument that more folks in academia would purchase Bookends if it included an auto-sync option may be stretching the limit of reasoning on why to implement the feature. Folks in academia (as I am) can tend to be rather particular about their favorite software tools. They will choose one or the other based on considerations that do not always include in-built automation, especially when the manual options that could be automated already are flexible and efficient. In this regard, let's also face it that the only productive macOS options for citation management are Mendeley, ReadCube Papers, Bookends, JabRef, and Zotero and that these tools have distinctively different approaches that go well beyond whether the app has automatic sync with a BibTeX file. The analogy here is saying the more families with children will buy a base-level Ford SUV rather than a base-level Honda SUV when the former would include an automatic, daylight/night-time sensing rear-view mirror rather than a manual one.
The issue I was addressing is not that auto-sync would bring potential users to the BE space. I was countering Jon when he said that there are only a small number of users that would benefit from LaTeX friendly features such as auto-sync. But back to the issue, I use Keyboard Maestro scripts to export a static folder to my BibTex file. Takes seconds. And if needed, nowadays with Apple Silicon (M1 chip) I'm able to export 11k references in less than 2 minutes.
Last edited by Nhaps on Sat Oct 02, 2021 10:59 am, edited 1 time in total.
Dellu
Posts: 268
Joined: Sun Mar 27, 2016 5:30 am

Re: [Feature Request] Auto-sync to a BibTeX File

Post by Dellu »

This is one of the most requested features in BE. Jon consistenly rejected it.

I really ask jon to reconsider this feature. I specifically ask you to look at Zotero's addon known as Better Bibtex (https://github.com/retorquere/zotero-better-bibtex). It is an open software. You can inspect the code, line by line if you like.

The user first exports the Bibtex file. Once exported, the addon updates the entries if there is any change. There is even an option in Better Bibtext to update the file:
- when the system is idle
- on every change
etc.
Image 2021-09-30 20-09-08.jpg
Image 2021-09-30 20-09-08.jpg (75.1 KiB) Viewed 26573 times
So, if one is worried about the speed, the On Idle would be a good option.

This is really useful feature. Please don't underestimate its value for many of us because it helps us eliminate a lot of post-processing; exporting every time new references are added etc.

In addition to the Translators, that is one feature which keeps me to go back to Zotero.
Jon
Site Admin
Posts: 10048
Joined: Tue Jul 13, 2004 6:27 pm
Location: Bethesda, MD
Contact:

Re: [Feature Request] Auto-sync to a BibTeX File

Post by Jon »

I don't get it at all. As pointed out, there are scripts that will do this on demand, for the full library or a selected group (if you don't want to use Biblio -> Bibliography manually). We made Bookends scriptable (a big undertaking) so that not every capability had to be hardcoded into Bookends. This is a perfect place to make use of that.

Jon
Sonny Software
DrJJWMac
Posts: 345
Joined: Sat Jun 22, 2019 8:04 am
Location: Alabama USA

Re: [Feature Request] Auto-sync to a BibTeX File

Post by DrJJWMac »

> This is really useful feature. Please don't underestimate its value for many of us because it helps us eliminate a lot of post-processing; exporting every time new references are added etc.

Pardon when I get a bit flustered here ...

I create a BibTeX file solely at the point where I am compiling my final document. Putting in the citations is the last step in the processing. In the meantime, I use citation keys stored in Bookends, trusting that, once I am done, the BibTeX file generated at the end from Bookends will be complete. I do not need BibTeX auto-export each time I add a new citation. I probably never will. In this light, convince me why I should change my approach and also join the call to have auto-export of a BibTeX file built in to Bookends. Explain to me what benefit I am missing in a way that I (as an engineer) can appreciate.

Why must export be automatic? Why must it be something that happens EVERY TIME NEW REFERENCES ARE ADDED? What makes having a built-in "automatic export" (not "auto-sync") so important compared to a) programming a keystroke to re-export the BibTeX file and typing that keystroke each time you exit Bookends or b) writing an AppleScript that runs in an idle loop to do the export for you?

Are you linking the BibTeX file dynamically to a LaTeX document that demands immediately updated citations from Bookends via BibTeX? Are you linking the BibTeX file to a DevonThink database that post-processes it to give you an immediate update on the word correlations while you add citations into Bookends? Is this being dumped to Obsidian to get an immediate, automatic, dynamic graph of something? Are you feeding the BibTeX file into Tinderbox with some magic that provides insights that absolutely have to be done in real time? In these cases, perhaps maybe I have missed out solely because I don't do any of these actions (and probably never will). Or perhaps what you really should be asking is whether Bookends can implement improvements in the direct dynamic links to your favorite real-time program without going through BibTeX?
--
JJW
iandol
Posts: 465
Joined: Fri Jan 25, 2008 2:31 pm

Re: [Feature Request] Auto-sync to a BibTeX File

Post by iandol »

At least in my case, I collaborate and generate my work-in-progess back and forth multiple times per day. I use Scrivener+Pandoc to build a DOCX or LaTeX>PDF depending on my collaborator. I add references to Bookends as I write, and so need to get an updated BibTeX file multiple times in a day when writing. My database is ~8000 refs and growing which get converted from BibTeX to CSL-JSON via another script that also protects the case of words that are important to my field and Bookends does not do that.

My Bookends-Tools includes an "auto-BibTeX" exporter. But due to bugs in Applescript (Jon looked at it and it is not his fault, so can't be fixed without Apple) it causes a large memory leak as the number of references exported increases, and it takes a quite some time for the export (Applescript is significantly slower than the native BibTeX export). I don't think there is a better way to trigger this using Applescript at least. So in the end I usually do this manually, which although simpler in a recent update still requires a few clicks and several minutes of thumb-fiddling. So the GUI twiddle, the export, plus the custom script to convert to CSL-JSON and title protection it can sink up to 7-9 minutes. If I update my draft 6 times in a day, it does add up, and it really disrupts my flow (I don't want to be distracted by non-writing fiddling). Pandoc is awesome in that regard, set it and forget it. Bookends (well specifically Bookends export of BibTeX) is the bottleneck to my writing flow.

Other apps do sync to a BibTeX file dynamically (i.e. it is a technically feasible). I nevertheless understand "reliable" sync is a non-trivial engineering challenge, and Jon has clearly stated this would be a significant development job, and it is his right to prioritise features (which Jon is incredibly active at delivering, no other software I own is updated with new features as regularly as Bookends is). Jon knows what most users are requesting or asking for, so he knows how many of us are markdown/LaTeX geeks. But for someone with a similar workflow to me, Bookends support of a one-way BibTeX mirror of our database would be a major practical advancement. If Apple ever fix their bugs (don't hold your breath), then this would be less of an issue...
joao
Posts: 58
Joined: Fri Jun 17, 2016 4:23 am

Re: [Feature Request] Auto-sync to a BibTeX File

Post by joao »

Jon wrote: Thu Sep 30, 2021 1:33 pm I don't get it at all. As pointed out, there are scripts that will do this on demand, for the full library or a selected group (if you don't want to use Biblio -> Bibliography manually). We made Bookends scriptable (a big undertaking) so that not every capability had to be hardcoded into Bookends. This is a perfect place to make use of that.
I think the issue here is with the term 'auto-sync'.

One thing that would help in this specific case, and perhaps even more in other use cases, is if Bookends could call scripts automatically (i.e. implement AppleScript actions). For instance, every time a new reference is added or a key generated (or perhaps every time Bookends syncs its library), it would call any AppleScript that happens to exist in a specific 'actions' folder. This would mean that after an initial long export to bibtex, subsequent syncs could be automatic and fast rather than on demand.

For my specific use case, it would mean every time I put a reference in a specific Bookends group, I could have my script that updates the pdf metadata with data from bookends and move the pdf do a different subfolder (to sync with an e-reader). There could be many more use cases though. At the moment one of the limitations is that any AppleScript has to be explicitly called on demand.

Just a thought.
Jon
Site Admin
Posts: 10048
Joined: Tue Jul 13, 2004 6:27 pm
Location: Bethesda, MD
Contact:

Re: [Feature Request] Auto-sync to a BibTeX File

Post by Jon »

Thanks everyone for a lively discussion.

@iandol -- I assume that any given paper you write with collaborators doesn't use all 8K references. It would speed things up considerably if you create a group (perhaps a smart group) for a paper and exported that as needed. I suspect your 7-9 minutes would be a log or more faster.

One thing I can look at is the time it takes to create BibTeX-formatted references. I'll see if there any improvements to be had there.

Jon
Sonny Software
Post Reply