[Feature Request] Cite As You Write Online (e.g. Wordpress, Medium etc)

[damo.clarky]

Hi Again Jon,

After being side-tracked for the past 6 months or so, and after the successful release of v13 of Bookends, I thought I would revisit this topic.

I am attempting to use Ulysses and Bookends together. But not just for exporting to MS Word to submit to publishers, but also to post directly online via Wordpress, Medium and other online services. Ideally, I'd like to be able to scan HTML with temporary citations as initially discussed in this topic. But I understand this is not an trivial undertaking.

So I've been experimenting for most of the day attempting to find a workflow that can achieve this with current features, but I am stymied at every turn.

Past experiments with scanning HTML were problematic due to the abundance of curly braces. So I have attempted to do the scanning against the markdown source in Ulysses. Here is what I have tried.

Scanning Markdown as a text document - using defaults
Using the default citation options from Bookends and adding temporary citations to my Ulysses document, I can then copy my Ulysses sheet to an external folder (because Ulysses uses iCloud storage API which does not expose a Filesystem interface outside of Ulysses). This causes some of the advanced Ulysses markdown to disappear, but that's a separate issue. From there, I can then use the Biblio -> Scan Document option from Bookends to scan the markdown text file.

As already known, the problem is it attempts to parse any instance of curly braces { } as bookends temporary citations, such as those related to code blocks. This would be okay except where bookends attempt to reconcile code blocks containing curly braces as temporary citations, it changes the curly braces to brackets ( ).

As a further test, I added some JSON containing curly braces to a MS Word (version 16.23) document and then also added citations using the Bookends toolbar option 'insert citation'. Again within the Bookends toolbar, I then selected 'scan document' and it still attempted to scan my JSON as citations and changed the curly braces to brackets in MS Word. This I image would be an issue for anyone writing a paper that includes curly braces for purposes other than citations.



Question: I am wondering whether there could be an option in the scan document feature that you can be selected that will leave any 'unmatched' temporary citations (i.e. those that aren't actually temporary citations) unchanged. The upshot would be that while I would receive a list of warnings about unmatched cases, the curly braces would remain unchanged and the document would remain correct.

Scanning Markdown as a text document - testing different citation delimiters
So I tried using some of the other delimiters that are on offer [] [[]] <<>> || etc. Sadly they all interfere with Markdown syntax, other programming language syntax or otherwise run into the limitations of Ulysses external folder markdown formatting. So I next tried the 'bibtex' delimiter option in the same menu.



The hope was that it would insert the temporary citation as something like this (using APA6th):

\cite{Rutherford et al., 2015, #15612}

Having \cite{ as an opening delimiter is much less likely to cause parsing issues by the document scanner, but I can't get it to work.

It didn't add the \cite in front - it looks the same as the {} delimiter option.

If I turn on Bibtex in that part of the settings, it works, but then it uses the bibtex key field. I don't want to use bibtex keys.

Question: Is it possible to have more options for temporary citation delimiters? Can custom delimiters be selected for instance? For my usage case, double curly braces would work well {{}} and are actually the standard option in Ulysses Markdown. In code, they can be separated by white space in most instances and still be valid code.

If anyone has any suggestions for workflows that they use to achieve this multi-publishing goal, I'd be most appreciative.

Cheers,
Damo.

[Jon]

Changing the temp citation delimiters to the final ones if the reference is not found was a choice made long ago because I thought that was better in cases where one of a group wasn't found (e.g. one of three in a citation group). That behavior could be changed, and the enclosing punctuation left alone, if people prefer that.

As for double curly brackets, that's actually difficult because of the way curly brackets are emitted in RTF.

Jon
Sonny Software

[damo.clarky]

Nothing is ever simple is it.

Changing the temp citation delimiters to the final ones if the reference is not found was a choice made long ago because I thought that was better in cases where one of a group wasn't found (e.g. one of three in a citation group). That behavior could be changed, and the enclosing punctuation left alone, if people prefer that.

That makes sense. Perhaps making it an option in the scan document dialog, or in the settings window would be the most appropriate solution? Those who rely on historical behaviour can retain it and those adversely affected by it can turn it off.

As for double curly brackets, that's actually difficult because of the way curly brackets are emitted in RTF.

Bugger.

Is it because braces are part of the RTF encoding syntax?

Are there alternate possibilities here?

Thanks Jon.

Damien.

[Jon]

We offer a number of alternative possibilities already, ones that I would have thought avoided problems with math formatting and markup. You or others can suggest another set of ASCII key pairings that would work in all situations.

Jon
Sonny Software

[iandol]

Damien: you may be looking for a bookends-only solution, but remember that Pandoc can be scripted to do much of what you want I think. I use it to go from markdown+citations to plain text, HTML, BBCode for forums, LaTeX, DOCX and a zillion other outputs. It uses the citeproc engine (same as iOS Bookends IINM, Zotero etc.), and Bookends can generate Pandoc temp citations as its input and the BIB file intermediate. I'm not sure about how Ulysses does markdown, from my memory it uses its own variant markup, but it may be similar enough? So Bookends+Pandoc certainly can go from markdown+temp citations ⇨ HTML+formatted bibliography: the queries thinking about a Pandoc solution are how Ulysses provides input, and how Wordpress handles the output.

[damo.clarky]

We offer a number of alternative possibilities already, ones that I would have thought avoided problems with math formatting and markup.

I know, that is what I thought too. Until I worked my way through them.

{} - Programming Language block delimiters
[] - Markdown links
~~ - MarkdownXL Raw Source
|| - UNIX Shell Pipe
[[]] - Confuses Markdown links
<<>> - UNIX Shell file redirection

You or others can suggest another set of ASCII key pairings that would work in all situations.

Given the various syntax domains such as mark-up languages, mathematics and programming, I doubt this is possible. But I wonder how likely it is to have someone writing across all those domains. Thus, being able to choose your own arbitrary delimiters would help for most circumstances.

If that isn't possible, then I kind of liked the delimiters being prefaced, such as \cite{} when using curly braces. This is much more specific and so the likelihood of a collision, while not impossible would be improbable for most users. This also aligns with bibtex syntax from what I understand. That was what I was hoping the BibTeX citation temporary delimiters option within the scan & bib settings area would do. But I couldn't get it to emit the \cite part, without enabling full support for BibTex, which then precludes MS-Word document scanning.

In fact, the more I think about it, the better I think that second option is.

What do you think Jon?

D.

[damo.clarky]

Damien: you may be looking for a bookends-only solution, but remember that Pandoc can be scripted to do much of what you want I think. I use it to go from markdown+citations to plain text, HTML, BBCode for forums, LaTeX, DOCX and a zillion other outputs. It uses the citeproc engine (same as iOS Bookends IINM, Zotero etc.), and Bookends can generate Pandoc temp citations as its input and the BIB file intermediate. I'm not sure about how Ulysses does markdown, from my memory it uses its own variant markup, but it may be similar enough? So Bookends+Pandoc certainly can go from markdown+temp citations ⇨ HTML+formatted bibliography: the queries thinking about a Pandoc solution are how Ulysses provides input, and how Wordpress handles the output.

Hey thanks Ian.

I was hoping to avoid the added complexity of using Pandoc, although I will keep it in mind. Ulysses uses MarkdownXL, but you can also customise and adjust what the syntax is.

However, for me, the broader issue is the overlap of markup syntax, plus the fact that a great deal of my writing will include programming code snippets which will collide with all the existing Bookends citation delimiters.

What are your thoughts on something like the \cite{} delimiter syntax? Can you see any issues that I may have overlooked?

D.

[Jon]

I don't like adding it as a general temp citation method because it's specific to BibTeX. It's also only used for text files, it doesn't work with word processors. But the solution is already available -- you can use it now in conjunction with the Key field. Bookends will automatically fill out that field for you for the entire library pretty much instantly. Select all and then Refs -> generate BibTeX key.

Jon
Sonny Software

[iandol]

What are your thoughts on something like the \cite{} delimiter syntax? Can you see any issues that I may have overlooked?

First off, I think the more basic idea that Bookends should not change temporary citations unless it has successfully matched would be a good start, and I wouldn't think it would cause any trouble to existing use cases?

Whether \cite{} is useful or not I think depends on how Jon's code scans the text: making a robust pattern matching algorithm that handles edge cases, even with modern regular expressions, is often less trivial than it appears to be. Personally I'd like Bookends to be able to handle Pandoc temporary citations [@bibtexkey] natively (like Papers used to), but doubt it is worth Jon supporting it in terms of numbers of users involved. So I use my own scripts to convert back to Bookends defaults when necessary, and leverage Pandoc for everything else...

[damo.clarky]

First off, I think the more basic idea that Bookends should not change temporary citations unless it has successfully matched would be a good start, and I wouldn't think it would cause any trouble to existing use cases?

I tend to agree.

Jon, is this something you would consider changing in an upcoming release?

I don't like adding it as a general temp citation method because it's specific to BibTeX. It's also only used for text files, it doesn't work with word processors. But the solution is already available -- you can use it now in conjunction with the Key field. Bookends will automatically fill out that field for you for the entire library pretty much instantly. Select all and then Refs -> generate BibTeX key.

If I use this approach, I can't scan MS-Word documents, only text-formatted documents. Unless I am doing something wrong. I can send you details of my settings.

Whether \cite{} is useful or not I think depends on how Jon's code scans the text: making a robust pattern matching algorithm that handles edge cases, even with modern regular expressions, is often less trivial than it appears to be. Personally I'd like Bookends to be able to handle Pandoc temporary citations [@bibtexkey] natively (like Papers used to), but doubt it is worth Jon supporting it in terms of numbers of users involved. So I use my own scripts to convert back to Bookends defaults when necessary, and leverage Pandoc for everything else...

It is difficult to contribute ideas without having a basic understanding of the underlying algorithms used, and I can understand Jon not wanting to reveal too much of his intellectual property.

Delimiter collision is a very common problem. I just had a read of this wikipedia article which does suggest some solutions. In particular, the following may be applicable:

ASCII delimited text
This one is a long shot given that they use non-printing ascii characters and some editors may strip these out.

Bracket Delimiters
This is the approach currently used, and determining an uncommon combination of brackets might deal with most usage cases. For example, <{ }> as a quick stab.

Configurable delimeters
This was where I was going with the {{ }} or \cite{} idea.

I'm an old perl programmer from way back and being able to use qq{} for example instead of " " allowed using literal double quotes when double quoting. Was a convenient and simple workaround.

I suppose it comes down to what constraints Jon faces in implementing something.

As it stands, and Markdown aside, even if I were to write using MS-Word and include programming code snippets, all the Bookends scanning delimiters collide with something from even the most common programming languages.

{} - Programming Language block delimiters
[] - Arrays
~~ - UNIX user home directory/one's compliment operator
|| - UNIX Shell Pipe, Logic or bitwise 'or' operator
[[]] - Nested arrays
<<>> - UNIX Shell file redirection/C++ stream syntax

Hope this helps.

D.

[Jon]

I've done some preliminary tests and think I can get this to work -- double grave accent

``temp cite``

Will that work for you?

Jon
Sonny Software

[damo.clarky]

I've done some preliminary tests and think I can get this to work -- double tilde

``temp cite``

Will that work for you?

Jon
Sonny Software

Hi again Jon,

Sorry for the delay in responding. I really appreciate your efforts to assist with my dilemma. Thanks Jon.

The double grave accent `` would probably work okay for programming (graves are often used as a string operator for interpolation of text (i.e. javascript) or inline code execution (perl, bourne shell etc) because I think it would be unlikely to see an empty grave-quoted string.

However, unfortunately it collides badly with markdown, where it is used frequently to denote a code spans.

As for double curly brackets, that's actually difficult because of the way curly brackets are emitted in RTF.

The double curly braces would be the ideal format that would work well with Ulysses/markdown, plus programming. With the exception of some niche templating languages, {{}} should not collide because whitespace is not significant in most programming languages. So writing { { (with spaces between) is still valid syntax for programming, but would not be matched by Bookends.

I think I understand the issues with RTF. I'll contact you directly via support on this one.

Otherwise...

Changing the temp citation delimiters to the final ones if the reference is not found was a choice made long ago because I thought that was better in cases where one of a group wasn't found (e.g. one of three in a citation group). That behavior could be changed, and the enclosing punctuation left alone, if people prefer that.

This might be the only way forward where mismatches can occur arising from delimiter collisions, but don't change the braces. Is this something relatively quick/easy to change?

Thanks again Jon.

Damo.

[Jon]

I don't know. But even if so Bookends would report missing citations, which would be confusing. And it wouldn't work with Mellel. And perhaps other issues that don't immediately come to mind.

Jon
Sonny Software