Many scientists need to create plots, edit images, write papers, etc. on a regular basis. All these tasks require dedicated software. Many colleagues spend thousands and thousands of dollars / euros / … on such programs, however, for many specialized applications, very suitable open source alternatives exists. Here I discuss my preferred tools to get the job done.
Most software that I am listing here is free to use and available as open source. There are some exceptions to this rule, and these are pointed out below. I am very much in favor of open source software and an open and honest statement on privacy and a company’s business model.
Since I work cross platform most of the mentioned tools are available for Linux, OSX, and Windows.
If you use great open source software or have some better alternatives you’d like to propose, I would love to hear from you!. You can find my contact information here.
The Gist
Basically, the straight up link collection without explanations or anything. More information below. Non-free software is marked with an asterisk.
- File manager: fman*
- Image editing for scientists: ImageJ
- Image editing - photoshop alternative: GIMP
- Image editing - vector graphics: Inkscape
- PDF annotations: PDF Studio*
- PDF stitching, cutting: pdftk
- Plotting with a graphical user interface: Veusz
- Python editor: PyCharm
- Python environment: yt-project
- Python & PyQt builder for compiled software: fbs
- Python plotting: matplotlib
- Synchronization of files: FreeFileSync
- Writing LaTeX: TeXStudio
- Writing Markdown: Ghostwriter
- Writing Notes: Standard Notes
Jump to…
Backup & File management
FreeFileSync
Everybody needs a backup! FreeFileSync makes it easy to synchronize your files to external hard drives, USB sticks, etc. This is not the same as having an incremental backup, i.e., a backup that lets you go in time and keeps different version of the same file. However, if you want to sync hard drives between each other, e.g., to carry files between a laptop and a desktop, and you don’t feel comftable using straight up rsync from the terminal, FreeFileSync is perfect for you. It is also amazingly fast and can save synchronization profiles, so synchronizing your files becomes easy and convenient.
fman
fman is a dual-pane file manager that is ideal for navigating and moving files around. Many keyboard shortcuts make your life easier. Furthermore, fman has a very powerful plugin API and these plugins are written in python.
Note that fman is not open source / freeware. It costs €18, which includes updates for the first year. Updates in following years (a subscription) will cost €12/yr. This seems like a fair price for a program that has so many capabilities. Furthermore, fman’s creator promises that if he stops developing the file manager, he will put the code as open source on github, which would allow the community to keep on developing it / fixing bugs. Have a look here; in the section “fman’s Open Source Promise” it is stated that:
If no commit is made to fman in more than 6 months, then it will be open sourced under a BSD license.
Image Editing
ImageJ
Every scientist dealing with images (not plots, see separate plotting category below) probably knows about ImageJ and if you use it you likely develop a love-hate relationship with it. Originally developed with support of the National Institute of Health and the National Science Foundation (both USA), ImageJ is a complete suite of image analysis and editing software suite. It is completely free and if you download the FIJI version (Fiji Is Just ImageJ), it comes preloaded with tons of plugins. Want to do particle tracking? There’s a plugin. Need to stitch images together, there’s a plugin. The best thing, most of these plugins and features come with their algorithms and instructions published in peer-reviewed articles thus making it possible to cite them.
Very powerful and worth mentioning is ImageJ’s macro language. The macro editor has an integrated help feature that supports you in writing macros. Furthermore, you can also “record a macro”, meaning that you click and run and the macro editor is filled with the text commands. This can be very helpful in guiding you in creating some small automation in your image processing routines.
All in all, ImageJ has many features that are useful and is mostly used in the biological sciences for microscopy image analyses. However, it can surely be used for other applications too.
Inkscape
Inkscape is a vector graphics editor. It is generally compared to Adobe Illustrator, however, it’s not as bloated and comes at a much better prize - it’s open source and free.
You have a pdf that you need to extract an image from: use Inkscape. Want to make a figure that will never look pixelated / a flow chart / a schematic diagram? use Inkscape. Generally, whenever you need a program that can deal with vector graphics (and can handle / import / place regular pixel graphics as well) Inkscape is the right tool to use.
The Inkscape standard format is svg: scalable vector graphics. Some of the best features of Inkscape are its powerful export machine, which makes it easy to transform images into vector pdfs or pixelated graphics of any size and resolution you want. An additional benefit is that there is directly a LaTeX to vector graphics plugin if you need to add an equation to annotate your figure.
GIMP
As Inkscape can be compared to Illustrator, GIMP is the analog to Photoshop. I wanted to mention it here for completeness, however, I do not use GIMP very frequently. However, whenever I have a pixelated graphic to edit / change something in, GIMP is there and can usually get the job done. It will generally take me a while, which however solely attests to my ignorance and not at all to the software.
A nice thing about open source software, not just with GIMP: There are lots and lots of tutorials and instructions online. Don’t be afraid to do a web search to find out what you need to know.
pdftk
pdftk is a very simple command line tool that can be used to arrange, re-arrange, merge, split, … pdf files. The command line tool is free and runs on Linux and OSX, I have not had any experience with the Windows version. The nice thing about pdftk is that it is very simple and straightforward. You can find some help online on how to put two pdfs together, etc. The program is fairly fast. To merge two pdfs (which is what I do most of the time with it):
pdftk pdf1.pdf pdf2.pdf cat output output.pdf
Interesting here: since pdftk is a command line tool, feel free to script your way around stiching together more complex documents and to interface pdftk from your preferred programming language.
Plotting
Veusz
A plotting program that is open source, cross-platform compatible is something that I couldn’t find for the longest time. I first used Origin, then Igor, looked at ProFit and tested it out… These were all not cross-platform, not open source, and expensive solutions. Finally, I discovered Veusz and have used it ever since.
When you first start Veusz you get an interactive help menu with a quick tour. Please look through it, it helps with navigating everything. I noticed very quickly, Veusz is written by a scientist who knows a lot about writing software - it was exactly the way I was thinking as well :) Thus, it immediately felt very intuitive with widgets, etc. It’s obviously written in PyQt and even uses some of the same language that you know if you ever wrote software within the Qt framework. But I’m nerding out here…
Veusz also comes with a very sweet collection of examples. Go to Help -> Example documents and you see a whole list of plots that are examples. You can also check out the examples here on the website. Here’s the first example from that page:
Most features you need come pre-loaded. You can also directly use LaTeX commands in all annotations and labels and even make 3d plots right off the bat.
One of my favorite features is that Veusz does not come with a very convenient data editor. However, it allows you to link the data files directly as a data source. For example: You have your data in a comma separated text file, you can now load the data into a Veusz plotting file and plot them. You want to update the data interactively: Just use LibreCalc or Excel to edit the file, save it, go to the Veusz data list and update from the data from the source. You’re all set and once you’re used to this workflow you won’t miss the interactive data management feature in the plotting software itself. If your data is, e.g., from a model run and you have updated values you can just hit update as well! This is very powerful and helps you to not copy data between different programs. One more advantage of Veusz over other software: the vsz files that are saved by the program can be read by a regular text editor! So if you ever have an issue with data files that cannot be found, you can always go into the text file and update the paths.
There’s also a suite of tutorials and help files online to get you started.
One caveat: If you are used to a very expensive program, e.g., Igor, don’t expect Veusz to contain all the fitting routines and special filtering methods, etc. Veusz comes with basic fitting, sure. These are perfectly fine enough if you just need a black box for fitting. In that case you probably don’t care what routines are used anyway. If you need more, you might have to implement it yourself, find another way to get the job done, e.g., in python and matplotlib.
matplotlib
Matplotlib, a module for python, is the ideal tool to plot if you want to write a script in order to make your figure. Highly customizable, matplotlib will make your plot look gorgeous. This is especially useful when you have a suite of simulations or need to make the same plot over and over again for various data sets. Sure, it will take longer to make one individual plot, however, it can be really time efficient once you have to do a certain figure over and over again.
If you’re programming in python, you most likely know matplotlib anyway, so I’m keeping this short. If you really don’t or you don’t use it often enough to be fluent, have a look at their tutorials page and use their search function. All the examples come with well commented code that you can download and try out for yourself in order to understand what it does. This is extremely helpful when making your own figure. And matplotlib figures really look great if you put a bit of effort in.
Fun fact: The logo for matplotlib is actually created in matplotlib. The code to make the logo can be found here. This gives you some demonstration on how powerful it can actually be.
Programming
I personally use python for pretty much everything I do. This programming section thus only contains tools to work with python. This is not intended to be a comprehensive list, but rather shows my favorites.
PyCharm
If you have programmed in python for a while PyCharm is the IDE for you. The community version is fully free and comes with most tools that you want. The one thing I wish was free in the community version is to edit code remotely via ssh, however, if you want this feature you’ll need to upgrade to the professional version. The pro version costs $199/yr for the first year, $159/yr for the second year, and $119/yr for all subsequent years. Honestly, in years and years of programming, I only wished a handful of times that I had the pro version. The community edition generally gives me plenty of features. There is a free trial for the professional version if you want to go that route. The following description is based on the community version.
PyCharm comes with full git and svn support. You can directly commit to github from PyCharm if you feel like doing so, however, it color codes the files that have been changed / are new since the last push / pull. You can also directly do a comparison of different versions, which sometimes is very handy.
Command completion can be very powerful. Furthermore, you can manage your virtual environments (or your environment) directly from the editor. This can be useful, especially if you write a requirements.txt file for your project and run it on a different / new computer. PyCharm will read this file and will help you to install all the requirements.
Shortcuts to run your software can be created and you can decide what you want to run where and have a menu with shortcuts to directly start it up. The editor also has a linter that helps you to keep your code neat and tidy, according to the python PEP8 formatting guidelines.
PyCharm comes with many more features, such as a refractor, safe delete to check for cross-dependencies, etc. If you’re new to programming, this might not be the editor for you, because the amount of features probably overwhelm you. However, if you’re a seasoned python coder give PyCharm a try.
yt-project
The python installation manager you use is completely up to you. I honestly use several different installations, from straight up python and installing everything by hand for very clean projects, to using anaconda, which I generally recommend to students who want to start coding python and come from a windows environment. You can also use miniconda, especially on a Linux or OSX operating system.
Whatever you do, do not use your system’s provided python installation! This cannot be overstated. OSX comes preloaded with some weird Apple python, you’ll just start getting many many problems if you use this out of the box.
A nice in between version between anaconda and installing everything from scratch is the yt-project. This is basically a conda environment that is built for analyses of volumentric data - whatever that means. yt is installed via bash script that allows you to specify individual modules that should be standard for science. So all together, a neat packaged script to install your python environment. Check it out, you’ll notice that yt loves you!
fbs
One of the major downsides of python can be the fact that it is fairly difficult to compile your software. Let’s say you wrote a GUI and want to give it to your colleague who does not use python: You will need to install a python environment, all the required packages, etc., or go through the long and weary process of finding out how to compile the software yourself.
fbs, which stands for “fman build system” (see entry for fman above) is written by Michael Hermann. He states that he wrote fbs in order to facilitate what I just mentioned: easy compilation and distribution of python. fbs is open source and free to use and distribute if you write open source software with it. Otherwise, you’ll need to pay a developer’s license (see the website for more information).
fbs very much facilitates writing code and GUI in python and PyQt and then build and deploy it. Have a look at the tutorial on github to see some of fbs’ power. Once you get into it, you will need to also read the manual and might want to have a preventative look at the troubleshooting section.
One of the most common problems that you can run into with fbs is that you do not have the correct dependencies installed. It is crucial that you use the correct dependencies, or you might get weird compilation errors. The best way to work with specific dependencies is to use a virtual environment for each specific project. Do not use your standard python installation with all its packages, you will run into issues this way! Also, if you work on Windows, have a look at the additional software that is required in order to use fbs.
Finally: If you have dealt with python for a while but have not written a GUI application (but want to start doing so), Michael’s book could get you started. It costs €29.95 and is well worth the money. You’ll get a DRM free pdf immediately and can start enjoying it. The book is very well written and gives you a great start into GUI programming with Python and PyQt. Together with fbs, you’ll deploy compiled GUIs written in python that your colleagues can use in no time!
Writing
Ghostwriter
Ghostwriter is a very well designed open source editor for writing in markdown. This blog post is actually written and edited in ghostwriter.
Ghostwriter has overlays that give you a markdown cheat sheet (who can remember all the things?), a table of contents, and stats of your document. It has different themes that all look good, and in general is a very clean and well done editor. It also has a live preview and comes with a distraction free mode.
Ghostwriter comes with what the authors call a “Hemingway Mode”, where your delete key will be rendered useless. I guess the idea is that you’ll be writing and not editing - a funny idea.
Unfortunately, Ghostwriter does not come natively for OSX. However, it can be built for MacOSX, you can find the appropriate readme file here. Note: I have not tried this out.
Standard Notes
Standard Notes is a note taking app that is cross platform and syncs between your various devices. Compared to all the other free ¨services” that are out there, the beauty of standard notes is that it is end-to-end encrypted and that only you can decrypt it (zero-knowledge provider). This means that nobody at standard notes can read your great ideas! This also means: if you loose your passwords, you have to come up with those ideas again.
I have tested standard notes on OSX, Linux, Android, and iOS and it works great. The basic version is free, the pro version, which costs as little as $2.48/months if you subscribe for 5 years ($149 total) comes with many extensions such as advanced editors (including a LaTeX editor!), a github push functionality, and the file save function, which lets you save files fully end-to-end encrypted to an online service, e.g., dropbox, or similar. Check out the extensions here.
This is a note taking app that respects your privacy, and respects your ideas as your own!
Even if you don’t have the extension version, standard notes also has a note taker independent service that let’s you share files with your friends (up to 50 MB). Their FileSend is free to use and end-to-end encrypted as well. Furthermore, it gives you only a limited time to keep the file around, automatically deleting it after this time span.
TeXStudio
Finally, when it comes to writing your manuscripts, theses, books, there’s only one way to write: LaTeX. If you don’t agree, I hope the rest of this article was useful. I will not go into discussion on why LaTeX is my perferred choice but will rather present you with my favorite editor.
I recently found the TeXStudio editor, which is very powerful. Sure, it could need some work on icons and theme support, but it has some very key features that I think are crucial.
The pdf preview can be decoupled from the main window and shifted to a secondary monitor. Furthermore, you can turn on / off synchronized scrolling between the preview and the source and synchronize your pointer between editor and preview (and vice verse). This is especially useful when editing.
On the left side the editor has a menu to display a current table of contents. Click on a section and the editor jumps there. This is especially helpful to navigate large documents such as proposals, books, and a theses. There’s no need anymore to split your work into separate documents and use the \include command.
Finally, TeXStudio comes with autocompletion and a very elaborate help menu that allows you to look for that math symbol you hardly ever use and can’t remember. All in all, a powerful editor that makes your LaTeX experience so much smoother.
This product is fully offline so there is no need to share your work with the world before its ready. For privacy reasons, I do not think that online editors that are not zero-knowledge, end-to-end encrypted should be a thing. Your files belong to you, they should be in your control, and you should be in charge of them.
Have fun with these programs and let me know what you think and if you have additions.
-k0dk0d
Update 2020-04-13: I previously recommended a PDFStudio from Qoppa software as a pdf viewer and editor. I removed this recommendation due to privacy concerns. PDFStudio is pinging the qoppa server on a regular basis, until I have further evaluated this software I cannot honestly recommend it any longer. Stay tuned.