[Feature Request] Cite As You Write Online (e.g. Wordpress, Medium etc)

[damo.clarky]

Hi All,

I am wondering how many others would like to be able to use temporary citations from Bookends (i.e. Cmd-Y from bookends) in articles they write online using their browser (e.g. Wordpress or Medium), and then within their browser, convert the temporary citations to the proper ones before publishing and then add a bibliography? Or in other words, cite as you write (CAYW) online.

I have included a brief 3min video demonstrating the concept with a very simple browser extension - you know, a picture paints a thousand words as they say. Hopefully my Australian English accent is comprehensible. The concept does require a low-level feature to be added to Bookends that would have to be implemented by Jon (explained below for those interested).

I am prepared to create a more robust and thorough browser addon to implement this feature, but it is only possible if Jon implements the low-level functions necessary. Quite rightly, Jon has indicated that there would need to be sufficient interest in this feature to justify the development time necessary for him to implement. Jon, please chime in to clarify if I have misunderstood anything.

So if you are interested in having the ability to CAYW in your browser, please indicate your support in response (or use the poll). If you have any subsequent suggestions or ideas on additional features, likewise please share.


Changes Necessary to Bookends
Effectively, it is necessary for Bookends to add to its API, a request that passes text input (HTML qualifies) and specifies a format style, and the response returns the scanned output, as per the current Bookends Biblio -> Scan Document ... menu feature. Initial discussions with Jon indicate that the most effective implementation might be to add this to the Bookends Server implementation as a HTTP POST. The proof-of-concept video mocks this functionality with a simple node.js server that listens on the same localhost TCP port as Bookends would. The browser addon demonstrated in Firefox would send the request to the Bookends Server, and then replace the selected text with the (Bookends scanned) response.

I have shared the proof-of-concept code via github for anyone who is interested in the implementation - you may be surprised at how simple the code is.

Thanks for your consideration.

Regards,
Damien.

[iandol]

Yes, sounds great. I would probably continue to use my own system for this personally[1], but as long as I can interface it from Alfred, then I think it would make a good addition to Bookends Tools too. I think the idea to use Bookends server is quite smart, it means any tool that can make a HTTP request can use this (browser extension, Alfred, commandline).

----
[1] I do all my writing using Markdown, including temporary refs, then use Alfred to trigger Pandoc to take any text selection and convert it into HTML, BBCode (for forums) or plain text, so I can use citations anywhere... It is more powerful than using a HTTP post, BUT it requires having Pandoc installed and set up, so is not plug-n-play.

[DarrenIngram]

I could see some occasional usage, whereby presently I cut and paste via textpad instead as an intermediary.

[Jon]

Having Bookends respond to a request to scan text submitted to it is something I would consider adding. But I would be reluctant to do this via Bookends Server. Server has a number of drawbacks that make it ungainly for this, including that it has to be set up by the user and that the library being accessed must be closed when accessed (i.e. not available in the UI). Likewise, parsing text files is not a problem, but I don't know if scanning HTML is possible without creating a whole new parsing engine -- it may not be necessary. Bookends can certainly generate HTML.

I think such a tool would be easier to make and be more generally useful as an AppleEvent, which can be triggered via the Terminal if you prefer.

Jon
Sonny Software

[damo.clarky]

Hi Jon/All,

Closing the library from which you wish to cite using the 'server' is a big caveat, which is rather unfortunate.

The Applescript event method generally would still be useful, although in the case of web browsers and excepting Safari, browser sandboxing generally prevents accessing the local system. So calling Applescript APIs from a browser addon won't be possible without an intermediary.

For those familiar with 1password, they solve this problem using what they call 1Password Mini. It is a little 'agent' that runs in the background that any browser with their addon installed can contact to communicate with their full client application (running or otherwise). Bookends could achieve the same ends through a similar intermediary to 1Password Mini. But again, this requires further work of you Jon.

An intermediary agent could be community created of course (like the node.js mockup I created in the original post) that bridges between the browser and the Bookends. But if there are too many additional steps to get things working, it is less likely to be used and more problematic to support.

Will give this further thought. If anyone else has some suggestions for bridging the browser and bookends, please do share.

Cheers,
Damo.

[iandol]

Yikes, yes, closing the database is a deal breaker for using Bookends server.

Here is another idea. Use a URL scheme like bookends://sonnysoftware.com/scan/url-encoded-text — this can be called from any service. Bookends scans the passed text and returns the result to the clipboard. This is less elegant than the full request/response, but shouldn’t run afoul of security settings. It would require either manual paste from the user, or some kind of polling of the clipboard, but I’m not an extension developer so have no idea how that would work…

[Jon]

If this were to be implemented, I'm leaning toward the notion that the AE would pass the paper (TEXT, assumed to be UTF-8 or RTF) to Bookends with the name of the format. Bookends would scan it with the following options on: insert hidden citations (if RTF), create bibliography (add at the end), try to resolve ambiguous citations = false, proofreading = false). Bookends would return the scanned paper with bibliography at the end. There would be no UI.

I haven't tried anything yet, and may not bother if there isn't sufficient support for this, so I'm not sure if it would work well or not.

Jon
Sonny Software

[damo.clarky]

If this were to be implemented, I'm leaning toward the notion that the AE would pass the paper (TEXT, assumed to be UTF-8 or RTF) to Bookends with the name of the format. Bookends would scan it with the following options on: insert hidden citations (if RTF), create bibliography (add at the end), try to resolve ambiguous citations = false, proofreading = false). Bookends would return the scanned paper with bibliography at the end. There would be no UI.

Jon, I think that would work just fine for these types of usage cases. Another consideration, web browsers aside, is that adding such an Applescript Event would automatically make it accessible to apps such as Ulysses (and other HTML/Markdown Editors), where the writing is UTF8 Text (Markdown) or RTF. It would be quite a trivial Applescript (added as a service, with keyboard shortcut) to pass selected text from the App to Bookends, and then replace the same text with the scanned result.

I did a little research last night to determine whether some sort of bridging agent (like 1password mini) could be trivially implemented. Something quick to build, easy to install and reliable in function. I am thinking that an Electron App, with a simple config UI, that sits in the menubar, launches Bookends if it's not already running, and provides a RESTful type API that bridges the AE interface that already exists in Bookends. Quite a few things line up here, such that it may not be a substantial undertaking to implement, and should be reasonably robust. Such a tool would make Bookends accessible to Apps that have limited Applescript integration (including browsers) at the very least.

If fact, more broadly, the concept could be interesting beyond Bookends. But that's another matter.

I haven't tried anything yet, and may not bother if there isn't sufficient support for this, so I'm not sure if it would work well or not.

Jon
Sonny Software

I'm not sure what the threshold is for sufficient support. On reflection, I may not have posted this topic in the best forum - perhaps the Applescript forum would attract a more relevant audience. But I am hopeful that the above specification would be a modest undertaking, given the functionality largely exists already, just enacted via a different event (AE rather than UI).

But I'll be quiet now and see what transpires.

Cheers,
Damo.

[Jon]

Just a thought -- it doesn't matter if the browser has robust AS capabilities or not. Even if it has xero scripting functionality You can use UI scripting to get the contents of the web page (Select All and Copy), extract the text from the clipboard, pass it to Bookends for processing, save the result as a text/HTML file, then tell the browser to load it (again with UI scripting). Given that browsers on the Mac have AS dictionaries (I checked Safari, Firefox and Chrome) it's quite likely even a better experience could be put together.

Jon
Sonny Software

[damo.clarky]

Hi Jon,

Just a thought -- it doesn't matter if the browser has robust AS capabilities or not. Even if it has xero scripting functionality You can use UI scripting to get the contents of the web page (Select All and Copy), extract the text from the clipboard, pass it to Bookends for processing, save the result as a text/HTML file, then tell the browser to load it (again with UI scripting). Given that browsers on the Mac have AS dictionaries (I checked Safari, Firefox and Chrome) it's quite likely even a better experience could be put together.

The dictionary for Firefox from what I can tell is just all the Standard Suite stuff. I threw together a quick JSX script below to test using the clipboard to pass data back and forth with Bookends. Sadly, the data captured with the clipboard is plain text, and does not include all the HTML formatting. When the script pastes back into the web page, all the formatting is lost.

I've included the code below, in case I have done something wrong. Not sure about Chrome or Safari. They may fair better.

Code: Select all

'use strict';

//--- GET A REF TO CURRENT APP WITH STD ADDITONS ---
var app = Application.currentApplication()
app.includeStandardAdditions = true

var seApp = Application('System Events')

//--- Set the Clipboard so we can test for no selection ---
app.setTheClipboardTo("[NONE]")

//--- Activate the App to COPY the Selection ---
var Firefox = Application("Firefox")
Firefox.activate()
delay(1)  // adjust the delay as needed

//--- Issue the COPY Command ---
seApp.keystroke('c', { using: 'command down' }) // Press ⌘C 
delay(1)  // adjust the delay as needed

//--- Get the Text on the Clipboard ---
var clipStr = app.theClipboard()
console.log(`clipStr=${clipStr}\n`)

clipStr += ' And the next sentence.' ;

console.log(`clipStr=${clipStr}\n`)

//clipStr = clipStr.replace(/[\r]/g,'\n') ;
//app.setTheClipboardTo("[NONE]") ;
app.setTheClipboardTo(clipStr);

seApp.keystroke('v', { using: 'command down' }) // Press ⌘V 

//console.log(`clipStr=${clipStr}\n`)

//--- Display Alert if NO Selection was Made ---
if (clipStr === "[NONE]") {
  var msgStr = "NO Selection was made"
  console.log(msgStr)
  app.activate()
  app.displayAlert(msgStr) ;
}

[Jon]

I'm sure there's HTML in the clipboard, too, but you have to specifically ask for it. But Bookends won't scan HTML (it will treat it as plain text, and I'm guessing there will be problems with open/close tags). But you could try.

Jon
Sonny Software

[damo.clarky]

Hi Jon,

I did a small experiment by dumping out a complete HTML document (home page of medium.com) and ran a scan pass on it with bookends.

The scan looked at every instance of open and close braces {} and attempted to convert them to citations.

When writing in a browser, it would be unusual for the authored component to include css, but I can't say it would never happen.

So implementing the function would work in many cases, but probably not all.

On a side-note, I do wonder what happens now when you use bookends to scan text files that contain, for instance, computer source code where braces are common?

Perhaps as a starting point, an AppleEvent could be added to perform the function as discussed. If it proves useful and gains popularity, then adding a DOM parsing library to Bookends, and parsing the text as HTML (no matter the encoding) would provide reliable results. There are many MIT-licenced c/c++/obj-c libraries for parsing HTML into a DOM that you could leverage - you know standing on the shoulders of giants.

I understand that there is likely going to be some higher priority issues to attend to with the newly released Mojave, but I am wondering (hoping) if you might agree to adding this AE function?

Cheers,
Damo.

[Jon]

At the moment all AE-related features are on hold, as we're contemplating a major reworking of how AE are handled.

Jon
Sonny Software

[damo.clarky]

Does this coincide with Mojave's new security framework around integrations using AEs?

Or was a rework on the cards anyway?

D.

[Jon]

Nothing to do with Mojave -- that just adds a confirmation layer for the user. AE work fine in Mojave.

Jon
Sonny Software