LyZ: LyX plugin for Zotero

LyZ is a plugin for Zotero, which is intended to make working with LyX/Zotero more pleasant.

Features

  • Inserting citations to LyX from Zotero.
  • BibTeX database automatically updated when citation is inserted.
  • BibTeX database can be updated when the references in Zotero are modified.Get Zotero
  • Custom BibTeX key format.
  • Unique BibTeX key check.
  • Custom BibTeX translators.
  • Synchronization of BibTex keys between LyX document, BibTeX database and LyZ.
  • Editing of master/child documents (the same BibTeX database) and multiple documents (different BibTeX database for each document).
  • Persistent association of LyX documents and BibTeX databases.
  • Support for group cooperation.

Installation and Settings

  1. LyZ works with Zotero 2.0 and later.
  2. Install LyZ  from Firefox Addons and restart Firefox.
  3. Open LyX and set LyXServer path (in LyX go to menu, Tools > Preferences... > Paths and set LyXServer pipe to): Windows users can use the default path setting in LyZ, i.e. “\\.\pipe\lyxpipe”. Linux and Mac users can use e.g. “~/.lyxpipe” and change the LyZ settings accordingly.
  4. By default all characters are escaped, e.g. š becomes \v{s}. Set extensions.lyz.use_utf8 to true in about:config to avoid escaping (in case you need to use non-latin scripts such as Chinese).
  5. Open document in LyX.

Adding and inserting citations

  1. Open Zotero, select citation and either from LyZ menu or from context menu (right click on the citation) do “Cite in LyX”.
  2. When you try to insert citation into a new document (i.e. document unknown to LyZ) you have to select BibTeX database. You will be asked to either create a new database or choose a database you have created before (by LyZ). Type the database name (*.bib extension is automatically added if you don’t provide it) and press OK. The citation is added into the BibTeX database you have just created and inserted into the LyX document. Your citation should appear in LyX document now.
  3. Insert the BibTeX database into LyX document. See BibTeX and LyX.
  4. Insert additional citations, new ones from Zotero, previously added citation can be either inserted through the LyX interface or from Zotero.
  5. If you modify the reference information in Zotero, use “Update BibTeX” from LyZ menu to update the BibTeX database.

Custom BibTeX keys

  1. Custom BibTeX keys can be set in LyZ > Settings dialog.
  2. The default format is “author year title”. Good alternative for the use with natbib package is format “zotero”, which will produce “human unreadable” BibTeX keys. It consists of library ID and zotero item key, e.g. 0_XXXXX. Format keyword “zoteroShort” will produce only XXXXX, which can be used when exporting from OOo.
  3. Key formats can be composed from keywords “author”, “year”, “title”, “zotero” and separators such as “_” (the following characters should be safe to use: a-z, 0-9!$&*+-.:;<>?[]^`|), e.g. “author year title”, which will produce authoryeartitle BibTeX key, e.g. tichy1988thefoundations. If you prefer the default format used in Zotero BibTeX export (tichy_thefoundations_1988), change the pattern to “author _ title _ year”. The keywords and any separating characters have to be separated by space. Characters in citekeys are escaped and unsafe characters are removed conforming to /[^a-z0-9\!\$\&\*\+\-\.\/\:\;\<\>\?\[\]\^\_\`\|]+/g.
  4. If you use custom BibTeX translator and want to use the cite keys generated in the translator, go to LyZ > Setting and switch creation of cite keys off.

Update BibTeX

Menu command “Update BibTex” will update BibTeX database of the active LyX document. Whatever changes made in Zotero will be reflected in the update, including BibTeX keys if the BibTeX key format or the relevant information (first author, title, year) have been changed.

When a single BibTeX database is shared among several authors, e.g. using version control system such as SVN and CVS, LyZ database is updated from local working copy of BibTeX database.

Only current LyX document will be updated. Multiple document update is implemented, but seems unreliable. Needs more testing…

Synchronization between LyX/BibTeX/LyZ

When Zotero reference is changed, BibTeX database as well as the LyX document can be updated. This is useful when nice BibTeX keys are used, such as ‘author year title’ and e.g. the title or the first author’s surname has been modified, which will result in modification of the BibTeX key. When the “Update BibTeX” is run, you should select to update the LyX document in case your modifications to Zotero references might affect your BibTeX keys. Before the update, a backup of the active LyX document is made (extension *.lyz). LyZ will save, close and then reopen your document.

Notes on usage

  1. Credits: Lyz is inspired by Lytero, which I wanted to improve, but ended up starting from scratch. Hence the new name. Thanks to Dan Stillman and mronkko at zotero.org forum. Justin Wood for BOM and escaping suggestions and donation!
  2. Always modify your references in Zotero. If you modify the BibTeX database manually or using other software, LyZ will overwrite all the changes when you run “Update BibTeX“.
  3. Keep your Zotero items unique. LyZ does not make an attempt to identify duplicates, they will be treated as different items and unique BibTeX keys will be created.
  4. When working with master/child documents in LyX, you will have to specify the BibTeX database for each child document. This will be the same file as specified for the master document. Unfortunately LyZ has no way of knowing the relationship between documents, so it has to be done manually.
  5. Old writing projects can be deleted from LyZ menu. When the record of BibTeX database is deleted, all associated LyX document records are also deleted. When LyX document is deleted, associated BibTeX database will remain untouched as it may be used by another document. The actual file won’t be deleted.
  6. BibTeX database is uniformly exported to UTF8 and all characters are escaped. Exporting to doesn’t seem to have any advantages. When your references contain extra characters such as the degree symbol, add \usepackage{textcomp} to your preamble.
  7. URLs in bibliography. Use bibliography style that can handle URLs, such as apalike, or add \usepackage{url} to your preamble.
  8. LyZ stores the information about documents and BibTeX databases in lyz.sqlite in the Zotero folder. If you backup Zotero with some synchronization application, you will notice the new file.
  9. When working with non-latin scripts such as Chinese, set extensions.lyz.use_utf8 to true in about:config to avoid escaping.

Source code

Please explore at https://github.com/willsALMANJ/lyz.

Problems, Ideas, Requests

Please follow these tips to find out the source of your problem:

  1. Run Lyx in debugging mode from commandline by (Linux and Mac users change accordingly)

    lyx.exe -dbg lyxserver

    Useful messages should appear whenever you send some command from Lyz. A useful message will contain words like ERROR, undefined etc.

  2. To send a command from Lyz, you simply use one of the functions, e.g. “Cite in Lyx” or use command line, viz. http://wiki.lyx.org/LyX/LyXServer.
  3. Also you can use a variant of the last version of Lyz, which has additional menu item “Test…”. This will allow you to send single commands to Lyx. Currently Lyz uses the following commands (code in the parentheses indicates usage):
    • server-get-filename (server-get-filename)
    • citation-insert (citation-insert:LYZ:author2010title)
    • buffer-next (buffer-next)
    • server-get-xy (server-get-xy)
    • buffer-write (buffer-write)
    • buffer-close (buffer-close)
    • file-open (file-open:path, where path is the response from server-get-filename)

    All functions Lyz has are patched together using these commands. The most trouble some seems to be the server-get-filename.

  4. Extract useful information from Firefox’s Error Console and from commandline where you ran lyx.exe -dbg lyxserver and send it to me or even better file a bug, ask a question or submit a request at https://github.com/willsALMANJ/lyz

 

When Zotero reference is changed, BibTeX database as well as the LyX document can be updated. This is useful when nice BibTeX keys is used, such as ‘author year title’ and e.g. the title or the first author’s surname has been modified.

192 comments to LyZ: LyX plugin for Zotero

  • Mattia

    Hi,

    thanks for this plugin, it’s really useful and I’ve used it with great satisfaction. Just one question: any chance of seeing Lyz working with Zotero StandAlone? It would be really great for people that do not want to be bound to use Firefox.
    Cheers,

    Mattia

  • Jack Tanner

    LyZ 2.0.2 sometimes sends a signal that makes LyX crash. It has to do with changing the key of an entry, e.g.,
    1. use the cite key: author year title
    2. create a Zotero item without a year
    3. cite it in LyX via LyZ
    4. add the year to the Zotero item
    5. cite it in LyX again

    LyX shouldn’t crash, of course, but I wonder if LyZ is being as robust as it could. One thing that occurred to me is that LyZ writes a Zotero item key to the top of the .bib file, and then a separate author-year-title key to the Bibtex entry, and it also stores this info in the sqlite db. I think Bibtex is supposed to ignore fields that it doesn’t know about, so maybe LyZ could just write a Zotero key to the Bibtex entry directly, and not even have the keys table in sqlite.

  • Petr

    Thanks Jack,
    I am actually working on it right now and upon a suggestion I am changing that part a bit, so it should not happen in the next release. I will make the new release (lightly tested) probably tomorrow or during weekend.
    As for the keys in the database: I don’t want to have Zotero keys in the bibtex entry, because I want to let people decide on their one cite keys. I am keeping track of the keys to alert users that they need to run “Update Bibtex” when their cite keys changed.
    But you rightly point out that Lyz is not robust in this respect.

  • Jack Tanner

    Petr, thanks for thinking about this. I think you missed my point a bit, though. I also like Bibtex keys to be nice, like author year title, but my point is that the Zotero key could go in a field other than the cite key.

    @book{milne1296winniethepooh,
    author = {Alan Alexander Milne},
    ..
    zotero = {0_2A2DPD9P}
    }

  • Petr

    Ah, OK. You are right that it could be written in the bibtex entry. I thought about this at the beginning, but it seemed to take quite a while to process that. Now I don’t need to look at the bibtex entries at all, just read the zotero keys from the first line of bibtex file. I don’t remember exactly, but when I was testing with large bibtex files, it seemed to take a lot of time to and freeze FF for too long… this seemed like a quick a easy solution. But tell me if you think I am missing something, that’s certainly quite possible :)

  • Jack Tanner

    Ah, so are the zotero keys at the top of the bib file written in the same order as the bibtex entries below them?

    Then how about this: write the zotero key both to the top of the bib file and to the zotero field in the bibtex entry. Whenever the entry is updated in Zotero, you can still quickly find it in the bib file using the information from the first line of the bib file. But before updating the actual bibtex entry, you can check that the zotero key matches. If it doesn’t match, then it’s the wrong entry, and the top line is screwed up. At this point LyZ would need to reindex the top line by the order of entries in the file.

    I’m still not clear – what’s the point of the keys table in sqlite?

  • Petr

    The second purpose for the keys table is to be able to have different cite keys for one zotero id. E.g. when you are collaborating with people that want to have different format of the keys. I find it more convenient (most likely faster) to read the table rather than to process text from bibtex file.
    Why would you rather have the zotero ids stored in bibtex entries?

  • Jack Tanner

    I would store the zotero ids both in bibtex entries and at the top of the bib file. If you only have the top line, you can’t be sure that, say, the third bibtex entry is the same as the third zotero id on the top line. It’s a validity check. The zotero id in the bibtex entry should be authoritative, and the top line just for speed of processing.

    In an ideal world, there should be no bib file at all. LyZ would register with LyX as the citation database. When publishing (e.g., via pdflatex), LyX would call out to LyZ with a list of citation keys, LyZ would send their full zotero entries back, and LyX would generate the bib file as a side effect.

  • Petr

    Actually, the order of the entries in the bibtex file is not important, as far as I know. Thus I can simply truncate the file and fill it with updated entries. Much simpler and definitely faster then text processing.
    As for your ideal scenario: nothing is stopping you to hack that 😉 Actually before Lyz I have looked into connecting to Zotero directly from Lyx, because I don’t like switching windows, but there are (were) some rather annoying challenges to call FF api from the outside.

  • Jack Tanner

    Oh, I didn’t realize the bib was already a throw-away thing. You’re right, truncation is the way to go. But if my FF profile goes awa… The Zotero db will get populated by syncing to my Z account. But if I now run Update Bibtex, I’m screwed – the keys table was empty in the new LyZ install, and you just truncated my bib file.

    You’re right, it’d be nice to hack on that ideal scenario… some day :)

  • Petr

    There only one cure for this: BACKUP^2. Eg. I am syncing my important stuff to dropbox (versioned with mercurial) and sugarsync (their 1 version + my mercurial versioning) + evening external hdd (10 versions + mercurial). Now I feel quite safe 😉

  • Jack Tanner

    You’re right, but my point is that I’d never realized that the FF profile contained this information. Ideally, it’d be stored together with the LyX document / bib file.

    A workaround would be to check a bib file against a Zotero database to see if the cite keys match.

  • Jack Tanner

    I just ran an Update Bibtex that resulted in updating a bunch of entries. The list of updates in a pop-up window was so long that the window didn’t fit on my screen.

  • Petr

    Good point, I will remove that.

  • greg

    hi, does the new, forthcoming version of LyZ works with zotero as a tab? That would be great!

  • Chris

    Hi Petr!
    just discovered all your updates, thumbs up!

    I read above it is possible to unintentionally truncate/cleare the .bib file, maybe lyz could keep a couple of old versions of the .bib file aroung? (e.g. .bib-[1-5].bak) And refuse to write out empty files?

    Where you able to fix the cite keys in utf8 .bib files yet?:
    AUTHORfunctionendsithtringreturnthis.substrthis.length-endsithtring.lengthendsithtring;YEARtitle
    (Is that string „functionendsit…;“ between the AUTHOR and YEAR a thing of the past?)

    • Petr

      Hi Chris,
      good idea about several versions of backup files.
      I’ll think about the unintentional truncation a bit more. I didn’t see it as an issue, because I am a bit overcautious with regards to my creations and I regularly backup all my backups before I do something dangerous like “Update bibtex” 😉
      Ad the utf8 keys: please give it a try and let me know if you still see some unexpected results. I kind of forgot what was that about… I tried it few times on mixture of Chinese and accented characters and it seemed to be working fine.
      Thanks

  • Chris

    Hi Petr,
    my lyx 2.0 just finished compiling and everything works fine with it. UTF8 gives nice umlauts, and the cite keys are clean.

    (There wasn’t even a hickup with updating the .bib, did you add special support for lyx documents under version control (rcs)?)

    Thank you, perfect!
    Chris

  • Petr

    Lets hope it will stay that way 😉

  • Russell J. Wilson

    The biggest problem I can see right off is that it is only compatible with Z 2.0, not the current 2.1.7

    or ahould that be 2.x?

  • Christoph

    hi petr, i tried your plugin and it seems very useful, so thank you!

    however, the bibtex keys that are generated are strange and not very useful. in the settings it says “author year title”.

    so for the book “an introduction to raytracing” by andrew s. glassner from 1989 i would expect something like “glassner1989anintroduction” … but i get “s.1989anintroduction”.

    can you give me a hint on how to fix this?

Leave a Reply to Seongsoo Cancel reply