Best practices to work with the Bib-sync feature

A place for users to ask each other questions, make suggestions, and discuss Bookends.
Dellu
Posts: 268
Joined: Sun Mar 27, 2016 5:30 am

Best practices to work with the Bib-sync feature

Post by Dellu »

I am sure many of you guys are much more advanced experts on BE and bibtex. But, there could be new comer who want to use the bibsync feature. For them, I have put some advises, on the best practices to use the new bibsync feature.

You can add your own, if you have new observations, or useful comments. You can also tell me if I have made mistaken notes.


1) Edit your the Bibtex format first:
- Modifying the Bibtex.fmt doesn’t make BE to update the output. Therefore, you have to make the format the way you want it to be. Clean it, make it as good as it can be before you synced your library.
If you find errors on the format, for example I find that on the @Misc type, the Keywords field was containing some weird characters such as `}`. I need to fix the format first, deleted the synced Bib file, and re-create the sync.

2) Sort your references before you sync: It is very important to put edited books and proceedings after @incollections and inbooks, if you use crossref. For that, the best strategy is to order your items inside BE library before you sync them.

- I only latter I learned that the sorting order I use in the library window is respected on the export as well. Therefore, you have use the “type” as the sorting criteria; and put it in descending order where Z comes first, and A comes last.

That way, the Books proceedings will come after , inbook and incollection types. That will generate a correct crossref between the books (parents) and their children.

Unfortunately, this method fails for @proceedings and @inproceedings. Therefore, you need to use another tool to sort them correctly, if you have many references of these types.


A second alternative is to use biblatex. If you use biblatex, sorting is automatic. Unfortunately, at least for me, using biblatex, the problems outweighs its benefits.

A third alternative is to open the bib file in Jabref or Bibdesk and save it. These applications are tuned to soft references to fit to the requirements of crossref by default.

The fourth method is using other tools such as bibsort and bibtool. If anyone is interested I can share my workflow using bibtool and hazel to automate the sorting of entries in the bib file. I have fully automated this that every time I synced inside BE (updated the bib file), Hazel will pickup and run bibtool over it so that the references will be sorted correctly.

3) Make the export format as minimal as possible . I especially recommend to avoid exporting the abstract field because abstracts imported from the web are messy. They may contain some weird characters that could disturb the bibtex. Keywords are also unimportant because they are not normally printed in the Bibliography. Anything field that is not going to appear in the bibliography need not to be exported.

Here is my format for the @article:

Code: Select all

`@Article{`u1`,`$
author = {$a}, $
title = {$t}, $
journal = {$j}, $
volume = {$v}, $
number = {$i}, $
pages = {$p–}, $
doi = {$u17}, $
url = {$z}, $
year = {$d}`}`¬
and, for the @incollection:

Code: Select all

`@incollection{`u1`,`$
author = {$a}, $
title = {$t}, $
booktitle = {$j}, $
chapter = {$v}, $
editor = {$e}, $
publisher = {$u}, $
address = {$l}, $
pages = {$p–}, $
crossref = {$u3}, $
doi = {$u17}, $
url = {$z}, $
year = {$d}`}`¬
This is based on the new improved format which ignores empty fields.

4) Put your synced bib file at the root of your tex installation; and forget about it.
In the mac, putting the bib file inside /Users/username/Library/texmf/bibtex/bib/ is a good practice.
https://pages.uoregon.edu/koch/texshop/ ... stuff.html

Wherever you are compiling your tex files, you don’t need to move the bib file with the project. Tex will automatically find the bib files once you put them in the right path.

5) Edit inside BE: not on the bib file. Even if modifying the bib file turn out cause no problem, it is best to edit inside BE to make the main library as clean as possible.

Any illegal character appearing in your bib file causing errors, don’t be tempted to fix in the bib file itself. Go to BE and remove it and update it. That is the way to keep our house clean.
Last edited by Dellu on Thu Jun 30, 2022 11:07 am, edited 7 times in total.
Jon
Site Admin
Posts: 10048
Joined: Tue Jul 13, 2004 6:27 pm
Location: Bethesda, MD
Contact:

Re: Best practices to work with the Bib-sync feature

Post by Jon »

Dellu wrote: Tue Oct 12, 2021 2:52 am Jon is improving these formats so that empty fields will not be exported. I am waiting for these improved formats to appear in the next release.
It was included in 14.0.1. Note that it's the new default BibTeX format. If you have previously edited it, your version will be in the Custom Formats folder and that's what Bookends will show/use. If you want to use the new BibTeX format (and perhaps make it the basis for a new custom version), you should delete the old custom format (or at least move it out of the Custom Formats folder).

Jon
Sonny Software
iandol
Posts: 465
Joined: Fri Jan 25, 2008 2:31 pm

Re: Best practices to work with the Bib-sync feature

Post by iandol »

For those of us who prefer to use Pandoc's CSL engine for referencing (even if we then compile to LaTeX), some additional tips:
  • Pandoc can compile more quickly if it uses CSL-JSON, therefore we can run a script to automatically generate the JSON from the synced BibTeX file.
  • In this case we can use a very cool tool, 'jq' to post-process the JSON, removing fields and compressing the JSON to further speed things up. My 14MB BibTeX file becomes a sleek 3.3MB JSON file and therefore loads and is processed much quicker.
  • This can be automated so that every time the bib file is synced, these commands are run and the JSON is automatically kept up-to-date.
The basic commands to create the JSON from the synced BibTeX:

Code: Select all

pandoc -f bibtex -t csljson SYNC.bib > Temp.json
And to process (remove any abstract/keyword or address fields) and white-space compress the JSON:

Code: Select all

jq -Sc '. | del(.[]."abstract",.[]."keyword",.[]."publisher-place")' < Temp.json > FINAL.json
If you customise your BibTeX.fmt you don't need to remove the abstract / keywords etc. fields but the jq command will work irrespectively and still remove extraneous whitespace.

These commands can be combined together into a script that can be run automatically to keep the JSON file up-to-date. There are several ways to run a script automatically. I use launchd (other options are automator, and commands like fswatch). To create a user launchd file, I use a great tool called LaunchControl to make creating and modifying launchd jobs easy. Launchd jobs can be triggered by a watched file to run a script. I can post a sample launchd file if anyone wants...
Dellu
Posts: 268
Joined: Sun Mar 27, 2016 5:30 am

Re: Best practices to work with the Bib-sync feature

Post by Dellu »

Jon wrote: Tue Oct 12, 2021 8:04 am
Dellu wrote: Tue Oct 12, 2021 2:52 am Jon is improving these formats so that empty fields will not be exported. I am waiting for these improved formats to appear in the next release.
It was included in 14.0.1. Note that it's the new default BibTeX format.

Jon
Sonny Software
Yes, I get it now. Thanks
DrJJWMac
Posts: 345
Joined: Sat Jun 22, 2019 8:04 am
Location: Alabama USA

Re: Best practices to work with the Bib-sync feature

Post by DrJJWMac »

Good, well-structured advice. Thank you.

I do not take 4) into practice. I absolutely prefer to have my bib files resident with the articles that I am developing. I am not in the habit of making one master bib file that is to be used by every journal article that I run through LaTeX. And, when I upload for publishing, the publishing site wants the bib file with contents that are only cited in the journal, not everything else.

WRT 6), the practice, if true, is rather distressing. I would hope that a change in the cite key itself in BE should not generate a new (duplicated) reference in the bib file. It should, at the export, overwrite the existing reference. My habit (as noted elsewhere) is to write and cite in a "free-hand" mode. I may end up having to change a citation key in BE (though I try to standardize the first time around).

All told, your practices for 4) and 6) apply best to folks who a) want auto-popup options to work as they generate their LaTeX document a) regardless of the document (i.e. the bib file must therefore be an actively updated master file of *all possible* citations, not simply a subset for the document at hand) and b) set a consistent approach to generating their citation keys every time they add a new reference.

Again, good stuff, just noting that some deviations can safely apply.

--
JJW
--
JJW
Dellu
Posts: 268
Joined: Sun Mar 27, 2016 5:30 am

Re: Best practices to work with the Bib-sync feature

Post by Dellu »

DrJJWMac wrote: Tue Oct 12, 2021 10:58 pm WRT 6), the practice, if true, is rather distressing. I would hope that a change in the cite key itself in BE should not generate a new (duplicated) reference in the bib file. It should, at the export, overwrite the existing reference.
That is actually true. I was also wondering why Jon doesn't use BE's internal reference number to identify the citation.

Code: Select all

@Article{Key,
%comment: Reference Number
% Refrence Number
author = {The author}, 
title = {The title}, 
journal = {Journal of XXX}, 
volume = {00}, 
number = {00}, 
year = {1999}}
But, I think his idea is to make the text clean. And, I agree with the decision. What we rather need or missing is a slightly advanced auto-generation method/fromat of Keys, just like the file renaming format we have in the Preference. Then, you don't have to change the keys manually. Having more than one library is a major issue right now because the keys will be duplicated across libraries (even if BE does a great job to create unique keys within a library).
All told, your practices for 4) and 6) apply best to folks who a) want auto-popup options to work as they generate their LaTeX document a) regardless of the document (i.e. the bib file must therefore be an actively updated master file of *all possible* citations, not simply a subset for the document at hand) and b) set a consistent approach to generating their citation keys every time they add a new reference.

Again, good stuff, just noting that some deviations can safely apply.
I totally agree. My approach is to have clean, large library where I can cite freely as I go. Then, I extract the references cited in a paper if the publisher asks for references.
Jon
Site Admin
Posts: 10048
Joined: Tue Jul 13, 2004 6:27 pm
Location: Bethesda, MD
Contact:

Re: Best practices to work with the Bib-sync feature

Post by Jon »

@Dellu

In my tests, if you change a citekey in Bookends and then sync, the existing entry is removed and a new one (using the new citekey) is added. There is no duplication.

Do you see something different?

Jon
Sonny Software
Dellu
Posts: 268
Joined: Sun Mar 27, 2016 5:30 am

Re: Best practices to work with the Bib-sync feature

Post by Dellu »

Jon wrote: Wed Oct 13, 2021 8:49 am @Dellu

In my tests, if you change a citekey in Bookends and then sync, the existing entry is removed and a new one (using the new citekey) is added. There is no duplication.

Do you see something different?

Jon
Sonny Software
Sorry, it is my mistake. yes, it is removing the old one. I was getting duplicated references at some point during the beta. This is wonderful. Thanks!
iandol
Posts: 465
Joined: Fri Jan 25, 2008 2:31 pm

Re: Best practices to work with the Bib-sync feature

Post by iandol »

Here is the launchd config file that monitors the BIB file for modifications and runs the script to protect my title case and generate a JSON mirror. Place it at /Users/YourUserName/Library/LaunchAgents/keep-json-sync.plist. In my case my BIB file is at ~/Dropbox/Papers/References/Core.bib — launchd watches this file for any changes, and if a change is noted, it runs the makeJSON.sh script automatically — you will need to change the paths to fit your locations:

Code: Select all

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>Label</key>
	<string>keep-json-sync</string>
	<key>ProgramArguments</key>
	<array>
		<string>/Users/YourUserName/bin/makeJSON.sh</string>
		<string>/Users/YourUserName/Dropbox/Papers/References/Core.bib</string>
	</array>
	<key>StandardErrorPath</key>
	<string>/Users/YourUserName/Dropbox/Papers/References/sync.log</string>
	<key>StandardOutPath</key>
	<string>/Users/YourUserName/Dropbox/Papers/References/sync.log</string>
	<key>WatchPaths</key>
	<array>
		<string>/Users/YourUserName/Dropbox/Papers/References/Core.bib</string>
	</array>
</dict>
</plist>
Here is the script that is called to generate the JSON file from the synced-bib file:

https://github.com/iandol/dotfiles/blob ... akeJSON.sh

This script runs the tool which protects the case in my article titles with a set of words related to my research field; you can comment out line 27 of makeJSON.sh to avoid running it (or modify that script to specify your own word lists). The result is that every time I sync from Bookends, I get an optimised JSON with protected titles. A log is output to sync.log in the same folder as the Core.bib file.

I modified the new BibTeX.fmt to remove abstract/keywords/address and also export full journal names (as Pandoc handles this separately):
BibTeX.fmt.zip
(2 KiB) Downloaded 348 times
Last edited by iandol on Mon Oct 18, 2021 12:32 am, edited 1 time in total.
iandol
Posts: 465
Joined: Fri Jan 25, 2008 2:31 pm

Re: Best practices to work with the Bib-sync feature

Post by iandol »

Here is an applescript (runsync.applescript) that should automatically run the re-sync command only if Bookends is running:

Code: Select all

if application "Bookends" is running then
	tell application "System Events" to set previousApp to name of 1st process whose frontmost is true
	tell application "Bookends" to activate
	tell application "System Events" to tell application process "Bookends" to click menu item "Sync Linked BibTeX File" of menu 1 of menu bar item "File" of menu bar 1
	tell application previousApp to activate
end if
This can be run every X seconds using launchd (in this case every 14400secs = 4 hours):

Code: Select all

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>Label</key>
	<string>run-sync.job</string>
	<key>ProgramArguments</key>
	<array>
		<string>/usr/bin/osascript</string>
		<string>/Users/ian/bin/runSync.applescript</string>
	</array>
	<key>StartInterval</key>
	<integer>14400</integer>
</dict>
</plist>
...this way you don't even need to remember to run the sync. The one limitation is that Bookends will become briefly focussed for a second when the script runs, which may disrupt your workflow. Maybe someone knows how to do this entirely in the background — ideally we would have an applescript command built into Bookends dictionary...
Nhaps
Posts: 244
Joined: Mon Sep 26, 2011 10:05 pm

Re: Best practices to work with the Bib-sync feature

Post by Nhaps »

Dellu wrote: Tue Oct 12, 2021 2:52 am - another alternative is to use biblatex. If you use biblatex, sorting is automatic. Unfortunately, at least for me, using biblatex, the problems outweighs its benefits.
Are you sure? What style are you using, Chicago? APA? BibLaTeX is much better than say, NatBib.
Nhaps
Posts: 244
Joined: Mon Sep 26, 2011 10:05 pm

Re: Best practices to work with the Bib-sync feature

Post by Nhaps »

iandol wrote: Tue Oct 12, 2021 11:35 am In this case we can use a very cool tool, 'jq' to post-process the JSON, removing fields and compressing the JSON to further speed things up. My 14MB BibTeX file becomes a sleek 3.3MB JSON file and therefore loads and is processed much quicker.
Speed is the only benefit? Sounds like it's a lot of work for that. Just trying to understand your workflow.
iandol
Posts: 465
Joined: Fri Jan 25, 2008 2:31 pm

Re: Best practices to work with the Bib-sync feature

Post by iandol »

Nhaps wrote: Sun Oct 17, 2021 8:15 pm Speed is the only benefit? Sounds like it's a lot of work for that. Just trying to understand your workflow.
Yes, speed is the only benefit; the script does the work automatically in the background (and I still need to protect my titles so I will always run a post-processing script anyway) — I have no idea whether the background "energy cost" is higher for pre-processing to JSON vs. Pandoc having to do slower parsing on each compile, but I tend to compile more than I update the JSON file so I suspect it is more efficient overall. From what I remember, Pandoc was around 2-3 times faster which was significant in terms of waiting for Scrivener to finish the compile...
Dellu
Posts: 268
Joined: Sun Mar 27, 2016 5:30 am

Re: Best practices to work with the Bib-sync feature

Post by Dellu »

Are you sure? What style are you using, Chicago? APA? BibLaTeX is much better than say, NatBib.
Yes, I was using some version of Harvard style. But, the main issue with BibLatex for me was the speed. The biber was about to burn my old mac. There are some other annoyances with Biblatex. I have written about it them before, https://tex.stackexchange.com/a/352078

Other disadvantages of the BibLatex (Biber) vs Bibtex

I recently moved to biblatex because of some weakness of the bibtex in relation to crossref. Unfortunately, BibLatex has its own.

1. Only the year will have the link (Why only the year is as a link in \cite reference(biblatex-apa and hyperref: right parenthesis not part of link for \textcite ; Why only the year is as a link in \cite reference )

2. some of the favorite style, the Harvard styles for example, are not natively available for BibLatex (Harvard Reference using Biblatex. You need to do a lot of manual work if you want get format like agsm.

3. Biber, the backend processor for biblatex is much slower (Why is biber so slow?)

4. Biblatex is not supported by many journals.

5. I have to use more extended texts like \textcitet in place of \citet. The same extensions are made on other default commands. The shorter the text, the better. In addition, spell checkers, even the latex aware ones, pick these extra commands as misspellings. That is another annoyance.

My final conclusion is, if your problem with bibtex (natbib)is minimal, stick with it. It is faster; and standard for submission.
larilela
Posts: 24
Joined: Mon Jan 21, 2019 12:47 pm

Re: Best practices to work with the Bib-sync feature

Post by larilela »

Thanks a lot for this helpful post! I've tried to follow your steps, but unfortunately – although I sorted the entries in my library window according to type,in descending alphabetic order – I don't get a bibtex file sorted by reference types (@article, @incollection etc.) In fact I cannot tell how it is sorted at all, it's neither alphabetic nor by types. What might I have done wrong ? PS: I marked all entries in the window and hit export -> selections -> bibliography format -> Bibtex.fmt -> bibtex MarkUp
Post Reply