Interesting...and not mutually exclusive.
Jon
Sonny Software
Request/idea: NoteCard browser
Re: Request/idea: NoteCard browser
@DrJJWMac @Dellu
I've thought about creating a one-way sync output for notes as Markdown, similar to how BibTeX is handled. But there are key differences that must be taken into account and resolved.
1. I'm envisioning this for notes, not PDF annotations. As I've explained in other threads, annotations belong to PDFs, not Bookends, so opening up thousands of PDFs, collecting the annotations, and outputting potentially 10's of thousands to a text file is an intensive and time-consuming operation (and who wants 10's of thousands of extracted annotations?). In any case, if for some reason you do want that, the Extract Annotations feature can do it for you with a few additional steps.
2. Notes would be output per reference, as they appear in the Notes field, as Markdown to a single .md file. Each reference's notes would contain a Markdown link back to the reference.
3. I'm thinking each reference's notes should have a header, too. I'm leaning towards the title of the reference.
Unlike outputting BibTeX, this may be fast enough so that the file can be created from scratch every time the user elects to update. If not, I'd have to add other markers to allow rapid parsing and comparison of the notes in the file and those in the database.
How does this sound?
Jon
Sonny Software
I've thought about creating a one-way sync output for notes as Markdown, similar to how BibTeX is handled. But there are key differences that must be taken into account and resolved.
1. I'm envisioning this for notes, not PDF annotations. As I've explained in other threads, annotations belong to PDFs, not Bookends, so opening up thousands of PDFs, collecting the annotations, and outputting potentially 10's of thousands to a text file is an intensive and time-consuming operation (and who wants 10's of thousands of extracted annotations?). In any case, if for some reason you do want that, the Extract Annotations feature can do it for you with a few additional steps.
2. Notes would be output per reference, as they appear in the Notes field, as Markdown to a single .md file. Each reference's notes would contain a Markdown link back to the reference.
3. I'm thinking each reference's notes should have a header, too. I'm leaning towards the title of the reference.
Unlike outputting BibTeX, this may be fast enough so that the file can be created from scratch every time the user elects to update. If not, I'd have to add other markers to allow rapid parsing and comparison of the notes in the file and those in the database.
How does this sound?
Jon
Sonny Software
Re: Request/idea: NoteCard browser
Dear Jon, I fully agree with your points number 2&3. It sounds great.
The idea was to leave it for the user which references to to have exported (synced) annotations. It doesn't have to be automatic and universal. Select a few references manually-->put them into a group-->sync-export the annotations (notes). But, I agree that this is tangential feature. You can leave it for now; and then you can think about it in the future.
Yes2. Notes would be output per reference, as they appear in the Notes field, as Markdown to a single .md file. Each reference's notes would contain a Markdown link back to the reference.
yes3. I'm thinking each reference's notes should have a header, too. I'm leaning towards the title of the reference.
Yes, it all sounds great.Unlike outputting BibTeX, this may be fast enough so that the file can be created from scratch every time the user elects to update. If not, I'd have to add other markers to allow rapid parsing and comparison of the notes in the file and those in the database.
1. I'm envisioning this for notes, not PDF annotations. As I've explained in other threads, annotations belong to PDFs, not Bookends, so opening up thousands of PDFs, collecting the annotations, and outputting potentially 10's of thousands to a text file is an intensive and time-consuming operation (and how wants 10's of thousands of extracted annotations?). In any case, if for some reason you do want that, the Extract Annotations feature can do it for you with a few additional steps.
The idea was to leave it for the user which references to to have exported (synced) annotations. It doesn't have to be automatic and universal. Select a few references manually-->put them into a group-->sync-export the annotations (notes). But, I agree that this is tangential feature. You can leave it for now; and then you can think about it in the future.
Re: Request/idea: NoteCard browser
That's an important point. I was thinking of exporting any reference with an entry in the Notes field, which mirrors what we do with BibTeX (any entry with a citekey is exported). That's the simplest option, requiring no setup and ensuring that all notes are available to the destination app (e.g. Obsidian). I don't want to make this too complicated, but there could be a dedicated static folder that holds the references, or Bookends could export the selected references. But realize that the new file would overwrite the old. So if you exported the entire library's references, then exported a single reference, the file would contain only the latter.Dellu wrote: ↑Sat Jul 01, 2023 3:08 am The idea was to leave it for the user which references to to have exported (synced) annotations. It doesn't have to be automatic and universal. Select a few references manually-->put them into a group-->sync-export the annotations (notes). But, I agree that this is tangential feature. You can leave it for now; and then you can think about it in the future.
I prefer the first option (export all references with notes).
As for the export of a subset of references with annotations, we have exactly that now: Extract PDF Annotations. It is *extremely* flexible and you can do what you want in 2 steps.
1. File -> Extract PDF Annotations. Click Extract.
2. File -> Save PDF Annotations. Click Save.
It also allows you to output a ton of information along with the annotations. It can be used to create an .md "sync" file of course, but it can do much, much more.
The new feature we are discussing would be much simpler and designed for a single purpose, to keep an up-to-date list of all references and their notes in an .md file accessible by other apps.
Jon
Sonny Software
Re: Request/idea: NoteCard browser
Thank you for bringing the Extract annotations feature.
I don't know when this was added. I just tried it now; works great. Thanks.
I don't know when this was added. I just tried it now; works great. Thanks.
Re: Request/idea: NoteCard browser
Jon:
Glad to hear about the progress on this.
1) As much as I would love the (automatic) sync to work with annotations as well as notes, I understand the limitations. Given also that we now have the option to collect annotations including notes, and given that Bookends can be scripted to run the collect + save operations ... with the push of a button as one might say ... I can't see any significant loss by not including annotations in the Note sync that you are developing.
2) + 3) As with the annotation export approach, would you be able to allow a user to choose a .fmt file to structure the output? Your efforts then need not concern these two questions, if only to the extent that you provide a sample NotesSync.fmt file for users to build upon.
One concern you raise is how best to manage the exported file when another app writes over it. If Bookends replaces the file, all the annotations from the other app are destroyed. I am not up to date enough on how other apps handle this issue. I defer to your finding for the best approach. This concern does however raise a potential feature request. Do we have a way to add stamps for the current date and current time in the .fmt file structure? If we are allowing Bookends to export annotations AND allowing other apps to manipulate content in the exported files, it would be nice to be able to keep track of who did what when. By example, if we accept dts as the format code for date+time stamp, I would include these lines for each reference in my NoteSync.fmt file
$#editor JJW@BE #createdon$ dts
@Dellu You might find insights from this posting at Obsidian.
https://forum.obsidian.md/t/workflow-ex ... dian/57867
Glad to hear about the progress on this.
1) As much as I would love the (automatic) sync to work with annotations as well as notes, I understand the limitations. Given also that we now have the option to collect annotations including notes, and given that Bookends can be scripted to run the collect + save operations ... with the push of a button as one might say ... I can't see any significant loss by not including annotations in the Note sync that you are developing.
2) + 3) As with the annotation export approach, would you be able to allow a user to choose a .fmt file to structure the output? Your efforts then need not concern these two questions, if only to the extent that you provide a sample NotesSync.fmt file for users to build upon.
One concern you raise is how best to manage the exported file when another app writes over it. If Bookends replaces the file, all the annotations from the other app are destroyed. I am not up to date enough on how other apps handle this issue. I defer to your finding for the best approach. This concern does however raise a potential feature request. Do we have a way to add stamps for the current date and current time in the .fmt file structure? If we are allowing Bookends to export annotations AND allowing other apps to manipulate content in the exported files, it would be nice to be able to keep track of who did what when. By example, if we accept dts as the format code for date+time stamp, I would include these lines for each reference in my NoteSync.fmt file
$#editor JJW@BE #createdon$ dts
@Dellu You might find insights from this posting at Obsidian.
https://forum.obsidian.md/t/workflow-ex ... dian/57867
--
JJW
JJW
Re: Request/idea: NoteCard browser
@DrJJWMac
You raise a good point about another app editing the file, in which case Bookends really shouldn't be allowed to overwrite it. I now think recreating a one-way "sync" to mimic the BibTeX file sync is the wrong approach. We already have the much more powerful and flexible Extract PDF Annotations. Why not extend that to Extract Notes and PDF Annotations? It would have the same interface we have now, with a new checkbox for "Notes" (or "Notecards"). You can output notes alone, annotations alone, or both. Since the file is saved via the usual Save dialog, you will be warned if you're overwriting a file you want to keep, in which case you rename the file you are saving. As I showed earlier in this thread, there are only a few steps involved, the friction is very low. This now seems like the obvious solution to me.
Jon
Sonny Software
You raise a good point about another app editing the file, in which case Bookends really shouldn't be allowed to overwrite it. I now think recreating a one-way "sync" to mimic the BibTeX file sync is the wrong approach. We already have the much more powerful and flexible Extract PDF Annotations. Why not extend that to Extract Notes and PDF Annotations? It would have the same interface we have now, with a new checkbox for "Notes" (or "Notecards"). You can output notes alone, annotations alone, or both. Since the file is saved via the usual Save dialog, you will be warned if you're overwriting a file you want to keep, in which case you rename the file you are saving. As I showed earlier in this thread, there are only a few steps involved, the friction is very low. This now seems like the obvious solution to me.
Jon
Sonny Software
Re: Request/idea: NoteCard browser
A BibTeX file is inevitably an immutable resource that could just as well be written in a read-only mode. One-way sync makes sense for this case.
Should you consider simply extending the Extract PDF Annotations option, I might offer these thoughts.
* While the process to Extract and Save may seem simple enough with only two steps, please consider whether you can eliminate the intermediate preview window in some way. Perhaps this could be done with the use of an additional checkbox on the Extract dialog box
[ ] Save directly to a file
or alternatively, eliminate the preview window by default unless specifically requested
[ ] Show preview window (before saving to a file)
* There is a difference between a PDF Note/Note and a Bookend Notecard (as per previous discussions). The former is already included in the Extract sub-menu as "Notes". The latter can be included in extracted output by using the "n" designation in the .fmt file. You may therefore already have what you need. Just clean up the terminologies and explain it better ...
-- rename the menu Extract PDF Annotations (Extract Reference Studies OR Export Reference Notes OR ...)
-- rename the format descriptor for "n" as a placeholder for Notecards (rather than Notes)
Should you consider simply extending the Extract PDF Annotations option, I might offer these thoughts.
* While the process to Extract and Save may seem simple enough with only two steps, please consider whether you can eliminate the intermediate preview window in some way. Perhaps this could be done with the use of an additional checkbox on the Extract dialog box
[ ] Save directly to a file
or alternatively, eliminate the preview window by default unless specifically requested
[ ] Show preview window (before saving to a file)
* There is a difference between a PDF Note/Note and a Bookend Notecard (as per previous discussions). The former is already included in the Extract sub-menu as "Notes". The latter can be included in extracted output by using the "n" designation in the .fmt file. You may therefore already have what you need. Just clean up the terminologies and explain it better ...
-- rename the menu Extract PDF Annotations (Extract Reference Studies OR Export Reference Notes OR ...)
-- rename the format descriptor for "n" as a placeholder for Notecards (rather than Notes)
--
JJW
JJW
Re: Request/idea: NoteCard browser
This is acceptable solution. One point I want to add is: separate Extractions for separate references: with the title of the reference as the title of the extraction...Jon wrote: ↑Sat Jul 01, 2023 4:21 pm
You raise a good point about another app editing the file, in which case Bookends really shouldn't be allowed to overwrite it. I now think recreating a one-way "sync" to mimic the BibTeX file sync is the wrong approach. We already have the much more powerful and flexible Extract PDF Annotations. Why not extend that to Extract Notes and PDF Annotations? It would have the same interface we have now, with a new checkbox for "Notes" (or "Notecards"). You can output notes alone, annotations alone, or both. Since the file is saved via the usual Save dialog, you will be warned if you're overwriting a file you want to keep, in which case you rename the file you are saving.
Re: Request/idea: NoteCard browser
@DrJJWMac
Bookends notes are real, notecards are virtual, a convenience. This is a significant distinction. What you see in the notecards is the separated paragraphs of the Notes field content. I try to distinguish between notes in the database and those in the PDF by referring to the latter as PDF notes. All that distinguishes them is how they are stored, and both are commonly referred to as "notes". Preview even conflates standalone "notes" that you have to click to see, and "notes" embedded in a highlight. I'm going to generally follow the conventions in Preview, which all Mac users have used at least to some extent, even if it's not their default viewer. In the export window I'd distinguish the type of "note". Maybe Reference Notes vs. PDF Notes. I'll think about that.
As you know, the window you are referring to is for configuration of the output, not a preview, and I'd be loath to bypass it under any circumstance. There are some options that you rarely change, but others might be changed frequently. As a case in point, if we add exporting of reference notes, you might want to output those to one file and PDF notes to another. If the configuration window was hidden how would you do that? You will suggest some workaround, such as holding down a modifier key forces the window to appear, but regular uses who inadvertently chose to hide the dialog will never know that and will think the output is fixed (I can tell you from experience that this will indeed happen).The configuration window can be accepted as is (which is your preference) by pressing Return or Enter, which seems pretty trivial. Something I could do to decrease the friction, however, is add keyboard shortcuts for Extract PDF Annotations and Save PDF Annotations, which would cut down on mousing.
@Dellu
As DrJJWMac pointed out, the format you use for the extracted content determines what information appears before the notes. If you want the title output, the Order field should contain simply the letter t. We include two formats whose names begin with Markdown. Create a new one based on one of those and modify it to output just the title.
Jon
Sonny Software
Bookends notes are real, notecards are virtual, a convenience. This is a significant distinction. What you see in the notecards is the separated paragraphs of the Notes field content. I try to distinguish between notes in the database and those in the PDF by referring to the latter as PDF notes. All that distinguishes them is how they are stored, and both are commonly referred to as "notes". Preview even conflates standalone "notes" that you have to click to see, and "notes" embedded in a highlight. I'm going to generally follow the conventions in Preview, which all Mac users have used at least to some extent, even if it's not their default viewer. In the export window I'd distinguish the type of "note". Maybe Reference Notes vs. PDF Notes. I'll think about that.
As you know, the window you are referring to is for configuration of the output, not a preview, and I'd be loath to bypass it under any circumstance. There are some options that you rarely change, but others might be changed frequently. As a case in point, if we add exporting of reference notes, you might want to output those to one file and PDF notes to another. If the configuration window was hidden how would you do that? You will suggest some workaround, such as holding down a modifier key forces the window to appear, but regular uses who inadvertently chose to hide the dialog will never know that and will think the output is fixed (I can tell you from experience that this will indeed happen).The configuration window can be accepted as is (which is your preference) by pressing Return or Enter, which seems pretty trivial. Something I could do to decrease the friction, however, is add keyboard shortcuts for Extract PDF Annotations and Save PDF Annotations, which would cut down on mousing.
@Dellu
As DrJJWMac pointed out, the format you use for the extracted content determines what information appears before the notes. If you want the title output, the Order field should contain simply the letter t. We include two formats whose names begin with Markdown. Create a new one based on one of those and modify it to output just the title.
Jon
Sonny Software
Re: Request/idea: NoteCard browser
I've added the keyboard shortcuts mentioned above for the next update. The process to extract and save without changing the configuration is just 4 key presses: Shift-Command-E, Return, Command-S, Return.
Jon
Sonny Software
Jon
Sonny Software
Re: Request/idea: NoteCard browser
As from past postings, I was confounded in making distinctions between the terms (PDF) Note, PDF Note, and (Bookends) Note. Your diligence to use either PDF Note (explicit) or Note (implicit for Bookends Note) is appreciated. Pedantic that I am as such, I might point out for example that the menu for editing PDFs (drop down from the gear icon) should say ... Add Text, Add PDF Note (rather than just Add Note), and Add (File) Tag.
I appreciated having the preview window to assure that I had structured my markdown .fmt file correctly. For my needs going forward, I should have little if any need to view or edit annotation + note content in a preview window. Perhaps my approach is not indicative of a typical user. In any case, I can create a KM macro or AppleScript to give me just one (or at worst two) keystrokes to collect + save annotation + note content to a file.
I appreciated having the preview window to assure that I had structured my markdown .fmt file correctly. For my needs going forward, I should have little if any need to view or edit annotation + note content in a preview window. Perhaps my approach is not indicative of a typical user. In any case, I can create a KM macro or AppleScript to give me just one (or at worst two) keystrokes to collect + save annotation + note content to a file.
--
JJW
JJW
Re: Request/idea: NoteCard browser
@Jon ... FWIW, the other app that I tested to manage PDF annotations is [ZenReader](https://zenreader.co.uk). It acts as a cross between Obsidian (as an electron app) and Highlights and LiquidText / MarginNote. It requires an on-line connection even to run the free version. Perhaps the videos that demonstrate how ZenReader handles annotation notes might spark an insight for your ongoing improvements to the notecard viewer in Bookends.
--
JJW
JJW
Re: Request/idea: NoteCard browser
Thanks for the heads up. I viewed the basic tutorial. Its features overlap with those of Bookends, but of course the goals and the way the results are displayed are different. As I've mentioned elsewhere, there are some very nice improvements coming to annotations and the note steam in the next update. Extracting metadata will be enhanced as well. I've refrained from posting too much in this thread because I want to follow, not steer, the discussion, and I've been happy with the result.
Jon
Sonny Software
Jon
Sonny Software
Re: Request/idea: NoteCard browser
Very cool features. I like that image annotation is supported. Tagging each of the annotations is very cool idea: very similar to the Coding feature in the QDA applications such as Atlas.ti. But, the latter take the tagging/coding further by adding concept maps and much more capabilities on the codes/tags. Incredibly powerful stuff. But, the biggest drawback with those QDA apps (for reading and annotating pdf) is that the annotation is database-internal. It can be exported separately; but the annotation of the pdf doesn't follow the standard-adobe annotation system.DrJJWMac wrote: ↑Sat Jul 08, 2023 4:55 pm @Jon ... FWIW, the other app that I tested to manage PDF annotations is [ZenReader](https://zenreader.co.uk). It acts as a cross between Obsidian (as an electron app) and Highlights and LiquidText / MarginNote. It requires an on-line connection even to run the free version. Perhaps the videos that demonstrate how ZenReader handles annotation notes might spark an insight for your ongoing improvements to the notecard viewer in Bookends.
Zotero is also going that way.
It is adding a lot of annotation capabilities; but, the annotations are internal to the app. If you open the same pdf with Acrobat-reader or some other reader, all the annotations are gone away. That is huge deferent to use those apps for annotating my pdf files. That is the hard lesson I learned from the late Sente.
Jon, you made really great decision in sticking to the standard pdf (Acrobat) annotations. We know there are some limitations with it; but it is really great that a pdf we annotated in BE can be further annotated by PDF-expert, Acrobat reader and many other apps.
I consider the Highlights app to have the most successful implementation of pdf annotation. It gives you the most freedom to annotate, write notes; and at the same time, the annotations are standard. That is amazing. BE is almost there. I cannot wait to see the new enhancements.