Feature request: Bib sync

[Dellu]

I use LaTeX and BE, and would second this request. What I do is drag the reference to a static folder, then use an applescript to export and update the bib file. My request is for automatic export to a default folder each time a reference is added to the static folder. This would eliminate the need for applescript

Hey Nhaps
A question: if you modified an entry inside BE; and re-exported it to your Bib library, will it overwrite it (using the unique id) or duplicate it? can you share the Apple script you are using?

[Nhaps]

I've explained multiple times why automatically generating a BibTeX file of all references with each reference addition/edit is a bad idea, and have explained how to do this on demand. I have no more to add to this thread.

Jon, even though you explained multiple times I think this crucial part of our workflow should not be sidelined indefinitely.

[Nhaps]

Hey Nhaps
A question: if you modified an entry inside BE; and re-exported it to your Bib library, will it overwrite it (using the unique id) or duplicate it? can you share the Apple script you are using?

Hey Dellu,
When you modify the reference in the main window, the copy in the static folder is automatically updated. They are actually one and the same. It is overwritten if you use the same bib name but if you change it then it is not, because it will create a second bib file. I can gladly share my Apple script but here are some caveats:

1. I am not versed in coding, this took me hours to get done. If you can improve on it that would be great.
2. You must fill in the fields of Biblio - Bibliography prior to using the script
3. You must select the static folder prior to using the script otherwise your whole database will be copied to the bib file.

Below you can find the script content. I saved it as an app onto my dock for easy access, next to BE icon. All you need is one click to deploy it and the bib file will be generated. Now that's nifty!

Code: Select all

tell application "Bookends"
	activate
end tell

tell application "System Events"
	tell process "Bookends"
		keystroke "h" using {option down, shift down, command down}
		delay 0.5
		click menu item "Mark All References" of menu 1 of menu item "Mark" of menu 1 of menu bar item "Refs" of menu bar 1
	end tell
end tell

tell application "System Events"
	tell process "Bookends"
		keystroke "b" using {shift down, command down}
		keystroke return
		delay 0.5
		keystroke return
		delay 0.5
		keystroke "r" using {command down}
	end tell
end tell

[iandol]

I thought this file watching utility may be potentially helpful for you guys:

https://github.com/emcrisostomo/fswatch

But looking at the database.BDB and database.BDB.JOURNAL files and they don't update when edits are made, so you can't use file changes to automatically trigger the update Applescript in the background. Perhaps BE uses a temporary file somewhere else that could be monitored?

[Dellu]

Hey Nhaps
A question: if you modified an entry inside BE; and re-exported it to your Bib library, will it overwrite it (using the unique id) or duplicate it? can you share the Apple script you are using?

Hey Dellu,
When you modify the reference in the main window, the copy in the static folder is automatically updated. They are actually one and the same. It is overwritten if you use the same bib name but if you change it then it is not, because it will create a second bib file. I can gladly share my Apple script but here are some caveats:

1. I am not versed in coding, this took me hours to get done. If you can improve on it that would be great.
2. You must fill in the fields of Biblio - Bibliography prior to using the script
3. You must select the static folder prior to using the script otherwise your whole database will be copied to the bib file.

Below you can find the script content. I saved it as an app onto my dock for easy access, next to BE icon. All you need is one click to deploy it and the bib file will be generated. Now that's nifty!

Code: Select all

tell application "Bookends"
	activate
end tell

tell application "System Events"
	tell process "Bookends"
		keystroke "h" using {option down, shift down, command down}
		delay 0.5
		click menu item "Mark All References" of menu 1 of menu item "Mark" of menu 1 of menu bar item "Refs" of menu bar 1
	end tell
end tell

tell application "System Events"
	tell process "Bookends"
		keystroke "b" using {shift down, command down}
		keystroke return
		delay 0.5
		keystroke return
		delay 0.5
		keystroke "r" using {command down}
	end tell
end tell

Thank you for your script. This is great.

[Nhaps]

SCRIPT UPDATE:

viewtopic.php?f=6&t=4212&p=19042#p19042

[naupaka]

Thanks for the discussion and ideas. I've put together an applescript to export a particular group or folder of groups from Bookends to a bibtex file. It's meant to be run from the command line, with parameters that allow you to select which group/folder you'd like to export as well as where you would like the exported .bib files to go (see comments at the top of the file for more details and syntax).

The idea is that it could be used as part of a makefile or other build script for a paper you're writing in TeX or markdown. If you wanted to, you could set it to be run from a cron job or via something like Hazel if you wanted automated refreshing of export for a particular group or folder of groups. I wouldn't recommend using it to constantly re-export your entire database, since as Jon said, that takes a while and is probably not usually necessary.

The script is available here: https://gist.github.com/naupaka/3637da8 ... 3575a7c2e1

[iandol]

Naupaka, this script is brilliant!!! As it uses proper Bookends Applescript events it is much more robust that simulating keystrokes, and you can get it to export multiple groups each to a separate BIB file. I was worried as I use unicode symbols in my group names it would fail but it handles partial matches well.

You should submit this as a new thread in the Applescript subform IMO.

Many thanks!!!

EDIT: I can reliably crash Bookends however if I try to use this to export all my library (~3000 refs)...

[naupaka]

Glad it is helpful!

I also can get it to reliably crash when I try to export my whole library of ~6500 references. Since the script will only work for references in a (static or smart) group whose name is returned from the

Code: Select all

«event ToySRGPN» given «class PATH»:"true"

syntax, I tried making a static group with all of my references and exporting that, but I get the error:

Code: Select all

bookends_to_bibtex.scpt: execution error: Bookends got an error: AppleEvent timed out. (-1712)

So it may be a limitation of how much data applescript can handle at one go (just a guess). I have been able to get it to work reliably with up to ~2000 references in a group, though, on my 2013 rMBP (takes almost a minute to export). Since ideally this script works quickly as part of a makefile for regenerating an Rmd/orgmode/LaTeX manuscript I sort of went in with the assumption that I would use it with a manuscript-specific group of references in Bookends.

[iandol]

I don't get any Applescript errors, only a Bookends crash, but yes this has to be an Applescript limitation as I can export the same references as bibtex without issue.

The bib file is empty until the applescript finishes (suggesting it is memory mapped until a final file write on line 164), and I wonder if you "flush" the file buffer to disk whether that stops the script from failing. Perhaps putting the write inside the tell loop on line 154? Trying to do anything in "easy" Applescript is anything but easy...

[iandol]

OK, I've made a fork of your script which no longer crashes Bookends, by writing to the file on every entry (I've added in some try catch statements too):

https://gist.github.com/iandol/90684eee ... 6762d5149e

I tested the speed and for 45 references it takes 2.5seconds compared to 2.3seconds for your original. It took just over 2 minutes to process my whole library of ~3000 refs, which isn't much longer than exporting from Bookends. During that time Bookends uses around 60% of 1 CPU and you can still interact with the UI...

We could possibly optimise this further by batch processing the references list (process 20 refs at a time), but I'm not sure how much we'd gain (how does one even debug the performance of Applescript)?

[Dellu]

This is getting interesting. Running a script from Hazel is very great idea.

1) newly added PDF in the attachment folder would trigger Hazel to call for this script (even more sophisticated conditions: like: if there are new PDF file in the folder + added time is not in the last 10 hours--to make good gaps to save processing power).
2) The script will fetch the references from a Smart group which collects references modified/added Today

[naupaka]

Nice! Good work @iandol. I merged in your edits and gave it a try on my machine - works to export ~6500 references with no issues, although it does take about 5 or 6 minutes to churn through them all.

I did discover another problem, though, which is that the file(s) is/are written using classic mac line endings, which are likely to choke commandline build tools for TeX. So I added in another section to use tr to swap the mac line endings for unix line endings after each file for each group is written.

New version incorporating my changes on top of your modifications at the same URL:
https://gist.github.com/naupaka/3637da8 ... 3575a7c2e1

Works well!

[iandol]

naupaka, I have updated your update to batch up the references (defaults to 25) which makes this a bit faster. Measuring using the time bash command I get about 0.2 seconds faster for 45 refs. But the bottleneck for this script is not the Applescript loop, but the Bookends «event ToySGUID» processing call which has linear behaviour (if you ask to 100 refs it takes 10 times longer than asking for 10), so trying to optimise the Applescript further is useless. But the batch mode is a bit quicker in general.

The line ending code was erroring as mv couldn't find temp.bib when I tried it so I just added $TEMPDIR so it is more robust against path differences.

Oh I also used octal codes for tr as I hate that Applescript converts \r and \n to real characters thus splits the line in the script. So \015 gets replaced with \012 and Applescript keeps the command on one line — sed may be better than tr as you can do inline editing and not generate the temp.bib file but that is for a rainy day...

[Nhaps]

naupaka, thanks for your new script, which is native to applescript (my script was a a lot simpler, keystroke-based).
iandol, your update sends the error: "Can't continue <<eventToySRGPN>>