A.I. integration?

[danzac]

Wondering about people's (and Jon's) thoughts on possible AI integrations in the future. Specifically, I've been introduced to ChatPDF (https://aieducator.tools/tool/https_www ... ator_tools) and impressed with how it can help interact with PDFs. It would be amazing to have this kind of AI technology easily accessible within Bookends in the future.

[iandol]

Hm, having recently played with a variety of large language models online and locally[1], I find them of limited use for interpretation of most topics in my field of study (Neuroscience). I'm sure they will improve to a certain extent, but I have ben shocked at how confident they sound for partial facts that are clearly wrong given the current consensus in my field. Things that are right are interspersed with things that are wrong in a coherent narrative I've shared these concerns with my students as I do worry about their use for education at present...

Technically, the online tools all use APIs so BE could certainly wrap some APIs in its browser, but the bigger issue at present is stabilty: things are moving so fast in this field tools around now may not be in two months time! I'm sure ChatGPT should be the most stable, though their API is not free and probably has a bunch of legalese Jon would have to deciper...

----
[1] gpt4all is a really nice locally run macOS tool with several free (but stripped down) LLM models to play with: https://gpt4all.io/index.html -- note these stripped down models (~13billion parameters) can't compete with the online ones (~150billion? parameters), though new models (see Orca from Microsoft research) are coming incredibly close...

[Jon]

I second iandol. I've played a bit with ChatGPT, asking basic science questions about topics if which I have a good grasp. The answers were impressive in their breadth but terribly disappointing in their depth. There is no actual cognitive ability (of course), and what you get is a hodgepodge of information without context. Remember, it collects its "knowledge" from the Internet, which we all know is always accurate and truthful. Garbage in, garbage out.

I tried ChatPDF, and what was spit back were comments in the summary like "Welcome to this helpful PDF file..." and "In this file, you will find valuable information about the use of...". Yikes. And the questions that were proposed led to summaries of sections of the introduction, results, and discussion (without any caveats, just the simple assertion that the answer is the "truth").

But to answer the question, it is obvious that generative AI is improving rapidly, and I imagine that it will prove useful in carefully controlled and standardized conditions, but by the time it becomes a reliable general purpose research tool we may all be interacting with it and each other via our Apple Vision Pros.

Finally, as iandol said, things are moving very fast, and today's bee's knees is tomorrow's AOL Online. Also, I don't really see why one would need any sort of formal integration. You can use ChatPDF within Bookends right now (I just did). Open the link in Bookends Browser and drag and drop a PDF attachment (all within Bookends, dragging the PDF proxy icon the the left of the PDF name in the PDF toolbar). The results are in Bookends Browser, and you can query them, copy the results, etc., all without leaving Bookends.

Jon
Sonny Software

[danzac]

Thanks for the responses all. It is helpful to know how I can potentially use ChatPDF in Bookends. In my mind, it was (perhaps in the future once it gets better), a simple way to ask questions about the specific PDF that is being read without having to upload or leave the reading space

[Jon]

Tools for generative AI will always have to do this via the Internet -- your local computational power and data storage is dwarfed by what is available remotely. Any such tools that could be used without Internet access would be primitive compared to whatever is the state of the art.

Jon
Sonny Software

[Nhaps]

I’ve been using Google’s NotebookLM with success, outside BE. It is an AI that works with uploaded PDFs and you have to create an account. It’s incredible what it can do (with the questions you ask)

[DrJJWMac]

> Remember, it collects its "knowledge" from the Internet, which we all know is always accurate and truthful. Garbage in, garbage out.

For the specific case mentioned, the AI is (presumably) parsing only within the journal article, it is not parsing through information on the Internet of things. So here specifically, the danger is less about whether any given journal article contains garbage. It is more that habits formed by using AI to collect lots of (good) knowledge over lots of (good) journal articles can pale in comparison to the training required to parse an existing knowledge base rapidly and efficiently in order to appreciate the larger context behind it as well as to gain confidence on the best directions to advance successfully from it.

The danger is about being given lots of chaff with perhaps a few kernels of wheat, and being left feeling good about getting lots of stuff so quickly. Alternatively said, the danger with AI is that it is mostly like fast food, giving you lots of sugar and carbs with little substance while drawing you back to another visit after the sugar high. Or by perspective from a few decades in training ... I can skim a journal article, collect the initial key insights that I need to understand where its report sits within my current understanding in the field of study, and decide the utility of going deeper in the journal article faster than I can feed that same article to an AI chat, ask questions, collect the answers, re-read its answers, sort out the chaff from the wheat, and figure out whether the AI report gives me any reason to dig deeper in the specific article itself.

All this being said, at some point, I will need to summarize specific content over a review set, such as to determine what relative fraction of journal articles in a set of a few hundred have specified a study using condition X. This is where I will be motivated to apply an AI tool. I do not expect that BE should provide me with an in-built ability to do use AI for this level of parsing.

[Mglo]

I have been able to supplement PubMed with ChatGPT4o to find relevant articles that I then evaluate. It has been able to highlight publications in response to specific and conceptual questions that would not be as readily found on PubMed. This has been useful and time saving. Indeed, it would be great if PubMed had a model trained on its corpus that could be used for this purpose.

Concerning Bookends, the one thing I was thinking could be a powerful addition is the implementation of Retrieval-Augmented Generation (RAG). I am not an expert on the details, but given the effort we have made to build and curate large databases of publications relevant to each of us, having a way to query those files specifically *might* be very powerful. In particular, any answer, I imagine, would be linked directly to the relevant sources. I'd be interested in hearing if and how that is possible. It seems manageable to implement it on the set of abstracts already in the database; what would be even more useful, but I suspect more difficult, to do so with a Pdf attachment folder that contains 1000's of pdf files.

Interested in the thoughts of Jon and others on the forum that have more of the technical possibilities.

[Jon]

Actually, we're waiting to see what sort of capabilities Apple is going to make available via APIs in the upcoming Sequoia OS. For many reasons we want to explore core OS capabilities rather than third party solutions. More information will come out as the OS betas progress. We will keep a close eye on that, and of course welcome insights from users.

Jon
Sonny Software

[iandol]

Concerning Bookends, the one thing I was thinking could be a powerful addition is the implementation of Retrieval-Augmented Generation (RAG).

I've used AnythingLLM: https://docs.useanything.com/introduction -- which is a local RAG alongside LMStudio (a local LLM runner). I created topic collections which I knew well and passed in a curated set of PDFs. Honestly, I found it a waste of time, the RAG answers just recapitulated stuff i already knew from curosry reading of the PDFs and didn't surface any novel insights I had missed. This is probably a limitation of RAG to overlay knowledge on a trained local LLM, I'm sure RAG will evolve but my take home is this was a waste of time. It maybe that online tools like NotebookLM remain the best place for this alongside professional fine-tuning...

Now the idea of using the Bookends database as a RAG source is intriguing, we would need an export format that a RAG tool could ingest, I may have a look over this. Again I don't think this is something Bookends needs to do, we have already existing tools and good automation features will allow us to chain Bookend > Applescript > RAG etc.

Actually, we're waiting to see what sort of capabilities Apple is going to make available via APIs in the upcoming Sequoia OS. For many reasons we want to explore core OS capabilities rather than third party solutions.

This makes a **lot** of sense, the landscape is still changing every week and so sticking to Apple's APIs makes good sense. As I mention above, we cn automate any other tools as needed...

[vinschger]

Yes, I think it's a good approach for Bookends to wait for the developer release of AI features/API in the next macOS version. I am not sure what AI could do for us in Bookends. Which features would you consider?

- Asking questions that are answered using the items in a selected group?
- Searching for PDFs?

I am a bit skeptical due to the potential for hallucinations but nevertheless interested in this new technology.

[joao]

I am not sure what AI could do for us in Bookends. Which features would you consider?

I have a simple answer for this. Organisation. Suggesting static or smart folders, tags, hierarchical folders, and yes, searching, locally on our library and remotely to find new references (and linking references to each other the way we can do now in bookends). All these tools are available in bookends, but we have to manually pick the references, tags, etc. Having an assistant do this for us, even if it’s not perfect, would make navigating the library simpler. It also seems to me that it could run locally as a small llm could handle this.

I would consider this to be quite useful. It’s certainly enjoyable to organise one’s library, but it takes too much time away from research.

[Jon]

That's an interesting idea, but I have doubts that an AI can guess how you would want your data organized, and everyone would be different. What I can imagine one could to would be able to ingest the metadata of the references you have and come up with commonalities and differences, and suggest grouping that that could make sense. But it seems to me that picking useful references and tags, as you say, requires some knowledge of the discipline and the way the data are to be used. As the AI APIs are rolled out and implemented we'll get a better idea of how they can harnessed to improve dealing with a personal reference literature. Suggestions like this one are a good place to start thinking about this, thanks.

Jon
Sonny Software