Copying PDFs to default Bookends folder when importing from XML

[nmadani]

When importing references from EndNote XML, is there a way that Bookends would copy the PDF attachments to its own default folder rather than maintaining links to several ad hoc folders that, especially with network drives, can be slow?
Thanks!

[Jon]

How are you importing the EndNote XML? If you are using File -> Import References -> From EndNote and Bookends can locate the PDF, it should be copied to the default attachment folder, in a subfolder called EndNote Attachments. If that's what you are doing and the PDFs are not copied, please contact tech support so we can see what's going on.

Jon
Sonny Software

P.S. Also note that there are several Global Change options that let you move or copy attachments in ad hoc folders to your default folder. This isn't a substitute for having Bookends copy them on import, but may be useful in any case.

[nmadani]

Thank you. Importing directly from EndNote is slow, and has now twice corrupted my EndNote database. Maybe it is an EndNote bug, but it happens.

What I am doing is exporting the EndNote library to an XML file before importing it to Bookends using:

File → Import References → From File or Clipboard ..., from where I choose from 'XML (From EndNote or Sente).'

I have found this process to be problematic. The default attachment folder has to be changed to ensure the PDFs are found when another XML file is to be imported, which then orphans the PDF attachments of previously imported references. There is no easy way to add 'Ad Hoc' folders that I found. It also seems that a multitude of ad hoc folders slows display of PDF attachments when Bookends has to sieve through them.

Meanwhile, it should be straightforward to infer the absolute paths of the PDF attachment if an assumption would be made that the XML file is in the same directory as the EndNote .enl file.

I think two additional options when importing XML files would solve my problem:
1. Bookends would ask for the root path of the PDF attachments.
2. Bookends would ask if it should copy any attachments found to its default folder.

Thanks again!

[Jon]

That's the first I've heard of that, but if EndNote database becomes corrupted it's definitely an EndNote bug. Bookends simply sends AppleEvents to EndNote to export the references and EndNote XML. It doesn't touch the EndNote database directly, but simply communicates with EndNote via their defined AppleScript dictionary. If would like me to look into this and can reproduce it with a small library, I'd be happy to try that here. BTW, the EndNote export of references consumes the large majority of the time when fetching them via, AppleEvents, importing the data is very fast.

In any case, although the code used to import a EndNote XML file is identical to that used for importing via AppleEvents, I see it doesn't import PDFs at all. I'm not sure what you were seeing, but what I found is that the import attachments flag wasn't correctly set and they were ignored. I've fixed that and now I see PDFs are imported correctly to the subfolder "EndNote Attachments" in the Bookends default folder. This will be included in the next update. If you'd like to try it before it's released, please contact us at support@sonnysoftware.com.

BTW, you can avoid the import menu and save a few keystrokes if you drop the XML file onto the Bookends library window.

Jon
Sonny Software

[Jon]

I may not have understand your comment about what is problematic. I think now that you mean the Global Change operations I suggested. In your case, it was "Move Attachments" option, which doesn't assume where the PDFs would go. Select the reference(s) and set the folder where you want them moved (in your example, it would be the default attachment folder). You can try it with a single reference if you like.

As for ad hoc folders, they are assigned by Bookends. When you attach a PDF from a non-default folder, Bookends adds that folder to the ad hoc list.

Jon
Sonny Software

[nmadani]

Thank you for your responses.

Let me try a different approach, starting with the task of non-destructively importing and unifying several EndNote libraries into Bookends.

EndNote libraries store PDF attachments in a subfolder of the data directory, labeled 'PDF', at the same level as the library's .enl file. Within this PDF subfolder, each citation has its subdirectory, which is named numerically to correspond to the citation's internal ID in EndNote. This serves to avoid name clashes when different PDF attachments have identical names, such as 'Reply', 'Corrections', etc.

Fast-forwarding to the "Global Change" function of Bookends that you suggested, it is destructive. In this move, the PDFs are relocated from their original locations. Importing data from another application should never compromise the integrity of that application's data. We can agree on this central tenet.

Using Apple events to extract citations individually is a slow and inefficient process. Especially when there are a large number of references, the process ties down two applications. It is far more efficient to export Citations from EndNote to XML, from which they can be subsequently imported into Bookends.

My first solution was to set Bookends' default folder to the data/PDF subfolder of the current EndNote library being imported as its EndNote XML export. This works until the following XML file is imported, requiring Bookend's default attachment folder to be changed to the new data/PDF location, which, of course, breaks all the previously imported attachment links.

Adding the attachments back manually and littering the Ad Hoc folder list in Bookends with countless entries defeats the entire purpose. Exercising Bookend's Global Change function after importing each EndNote XML file destroys the integrity of the original EndNote library.

I do not wish to sound harsh or offensive. I think Bookend's approach to handling PDFs is a software antipattern. I can imagine how it evolved to its current state by working around an early design choice that, while simple, was not flexible enough for efficient management of multiple libraries. To copy or move EndNote libraries, copy or move the file and folder. That is not the case with Bookends.

Perhaps the easiest solution would be for Bookends to optionally allow storing an absolute path to the PDF attachments, wherever they reside on the hard drive at the most granular, citation level, and leave the responsibility of not removing the attachments to the user.

Regarding the EndNote database corruption, we corresponded about it via email a couple of years ago when it first occurred.

[Jon]

Fast-forwarding to the "Global Change" function of Bookends that you suggested, it is destructive. In this move, the PDFs are relocated from their original locations. Importing data from another application should never compromise the integrity of that application's data. We can agree on this central tenet.

The global operation is NOT destructive unless you choose it to be. The dialog has a popup menu with the choices of Move and Copy. Choose the latter. This is documented, of course, and the popup is clear, but I'll change the menu to Move/Copy so it is not overlooked during casual inspection.

Using Apple events to extract citations individually is a slow and inefficient process. Especially when there are a large number of references, the process ties down two applications. It is far more efficient to export Citations from EndNote to XML, from which they can be subsequently imported into Bookends.

Again, you make assumptions that are not correct. The AppleEvent does not extract the references individually. Rather, it tells EN to output all references (or the selected ones, if you prefer) at once and send them to Bookends. You can examine the EndNote AppleScript dictionary in Script Editor yourself and see how that works.

Actually, I find it to be very fast and convenient, at least when dealing with even thousands of references. I tested this just now with an EN library with 2188 references -- it took < 5 seconds! In my case only a few had attachments, and it would have taken longer if there were 2000 of them. But copying files from one location to another (NOT moving) is going to take the same amount of time regardless of how it is done. I realize that your library is exceptionally large, and I didn't try this with 100K references.

I don't know if you realize this, but the importing via AppleEvents and via an exported file is schematically identical. In the first case, the AppleEvent tells EN to do the export and pipe the XML to Bookends, in the second you manually export and then manually import.

I do not wish to sound harsh or offensive. I think Bookend's approach to handling PDFs is a software antipattern. I can imagine how it evolved to its current state by working around an early design choice that, while simple, was not flexible enough for efficient management of multiple libraries. To copy or move EndNote libraries, copy or move the file and folder. That is not the case with Bookends.

I don't think you understood my reply, which was:

I see it doesn't import PDFs at all. I'm not sure what you were seeing, but what I found is that the import attachments flag wasn't correctly set and they were ignored. I've fixed that and now I see PDFs are imported correctly to the subfolder "EndNote Attachments" in the Bookends default folder. This will be included in the next update. If you'd like to try it before it's released, please contact us at support@sonnysoftware.com.

To rephrase -- the failure to copy the EN PDFs to the Bookends default folder when you import an EndNote XML file was a bug. I fixed it, and the fix is coming in the next update. And since I wrote that, I've been adding other capabilities that will make moving references/PDFs from EN to Bookends even easier. If you want to see how it works, the offer stands.

As to the claim that using AppleEvents corrupts and EN library (how could it?), I asked before for proof, meaning a reproducible example, not an anecdote. If you can reproduce a problem of course I will look into it.

Jon
Sonny Software

[nmadani]

Thank you again for your response. You are absolutely correct about the global change that allows copying rather than moving. I totally missed that.

Unfortunately, I could only transfer 236 references in 21 seconds using the direct import you suggested. I assume EndNote must be the rate-limiting step. I could live with that speed. It just takes longer.

Meanwhile, this solution led to other problems: I can import one EndNote library per Bookends library. When I try to import another EndNote library, nothing is imported. I can only reimport the EndNote library already imported. I restarted both programs with the same result. I even rebooted, and it still failed. I tested this using EndNote 21 and EndNote 25.

I ended up importing the second EndNote library into a new Bookends library to merge the two. However, while the EndNote library contained 283 citations and I selected all references to be imported, it only imported 235.

I have to assume that the problem lies with EndNote, and not Bookends. However, I doubt that I can present this to EndNote as a bug and receive any response other than this is not a supported feature.

My only option remains exporting to XML using EndNote, then importing into Bookends. The global attachment copying and changing the default folders is a chore, but works.

Thanks again for your time and efforts. This is why I am a Bookends customer, despite being forced to use EndNote due to collaboration requirements. I also look forward to the updated version.

[Jon]

On the EndNote import from more than one library in the same database issue -- I just tried this and had no problem (EndNote 21). There is one circumstance that you would see this, but I doubt this is the case. If Bookends can uniquely identify a reference (e.g. my DOI or PMID), it won't import it twice. So if the two sets of references where the same, they wouldn't be imported into the same library via EndNote XML. As I said, this is unlikely because you would know that they were the same sets.

Another circumstance is if both libraries were open in EndNote at once. Bookends tells EndNote to send the references in the "first document", which is the one on top in z-order. If you selected references in the second but then inadvertently brought the first library to the front, the import would be from the first library again, and as above, nothing would be imported because the references are already there.

Please don't use the global ad hoc PDF harmonization trick, it wasn't meant for this use and I agree is cumbersome. As I said, the next update, which isn't far off, will handle this for you transparently when you import an EndNote XML file. It will be no added effort at all.

Finally, if there is a problem in Bookends with references apparently not being imported (235 when you expected 283), I will of course address it. If you send me the database (without the PDFs to make it more manageable), I'd be happy to look into it.

Jon
Sonny Software