Pumilio

A Web-Based Management System for Ecological Recordings

Important note

Due to changes in my career path, this project is no longer under development. It may or may not work in newer systems. Feel free to fork it and change it and expand as you see fit.

Version 2.7.6 is out!

Citation: Villanueva-Rivera, Luis J., and Bryan C. Pijanowski. 2012. Pumilio: A Web-Based Management System for Ecological Recordings. Bulletin of the Ecological Society of America 93: 71-81. doi: 10.1890/0012-9623-93.1.71

Pumilio is a free and open source PHP/MySQL application for the management of sound archives and the visualization and manipulation of sound files. The system allows the user to load sound files in many formats, see the spectrogram of the sound, select regions of the sound for further analysis and insertion in a database, filtering, and many other manipulations.

All that is needed to browse and navigate the archive in the application is a modern browser with JavaScript and either Adobe Flash or HTML5. This means that it will work with the iPad, iPhone, iPod Touch, and newer versions of current browsers without Flash installed. The application has been tested with all recent versions of Firefox, Safari, Internet Explorer, and Chrome.

The concept of this application is to use a graphical front-end to edit, manipulate, convert, visualize and extract information of sound files. The code uses several scripts and command-line programs to achieve its purposes.

The major intended audience of the application is bioacoustics research groups. Other groups that may be interested include people interested in sounds, musicians, wildlife and nature sounds fans, and many more.

Have ideas, suggestions, or want to contribute code? All help is welcomed, just send an email to ljvillanueva@coquipr.com or using the Github system.

Open Data and outreach

The application was built to allow the administrators to select one of many ways of hosting the data online. All or none of the data can be made public:

Download the files directly on the browser, for both desktops and mobile devices
In R using the pumilioR package
With other applications using XML

In addition, the system has an export feature that makes it very easy to transfer the sound files, and the metadata, to another system. This feature can also be used for backups or to prepare data for posting in online data archives.

The application can be used to further outreach efforts by sharing data with the public in an easy-to-use system that can be navigated using Google Maps. Interested parties can listen to the sounds recorded using just a web browser, without requiring any particular software, plugin or operating system.

R package

A new package for R, pumilioR, is available from CRAN. It allows the user to query a Pumilio system, version 2.6.0 or newer, and get the data as a dataframe.

Check the website for details: http://ljvillanueva.github.io/pumilioR

Tweets by @PumilioDev

Important note on future development

The next version (v3) will contain many changes, including using a different CSS library, audio players, and background processes. Version 2.x is frozen, unless there is a bug to fix.

Change history

The last release to date is 2.7.6 (3 Apr 2015)

New for release 2.7.6:

Fixed a bug in the python script that adds files where it would insert files already in the database. Issues #33 and #30.

New for release 2.7.5:

Fixed a bug that did was not allowing the server to delete old temporary files.

New for release 2.7.4:

Fixed Issue #29 where the search function did not limit the results to the sites and collections queried.
Added fix on Issue #28 for marks that fell outside the range of the space shown.
Fixed Issue #20 where the script that adds files to the databases exited prematurely.
Fixed Issue #19 where the browse_site_data page had a broken link that returned an empty page.
Changed the way files are imported and added further checks to avoid problems with processed left hanging, related to Issue #16.
Fixed Issue #14 where the bits of the file were not queried and a default value of 16 was inserted in the database.
Fixed Issue #12 where two forms in the admin page were broken.
Several other stability and bug fixes.
Many thanks to users aloboa, kdarras, ecoinfo, and others for providing detailed information to fix these and other bugs.

New for release 2.7.3:

Fixed the form for search in the the main page.

New for release 2.7.2:

Updated the libraries JQuery, JQueryUI, Mobile_Detect, JQuery Crop, and JPlayer. Removed some outdated libs.
Removed some unnecesary features to simplify the code.
Reorganized the file details page.
Removed the optional variables from the default config.php file (config.php.dist).
Removed the readonly attribute from frequency and time fields in viewer.
Many stability and bug fixes.

New for release 2.7.1:

Mostly stability and bug fixes.

New for release 2.7.0:

Edited the xml page for the R package pumilioR v 1.2.
Moved some database calls to the abstraction layer.
Fixed a number of bugs.

New for release 2.6.1:

Reconfigured options of the spectrogram images, in particular adding a way to change the FFT window.
Moved some database calls to the abstraction layer.

New for release 2.6.0:

New version of the xml file to allow query from pumilioR.
Fixed a number of bugs.

New for release 2.5.4:

Updated a deprecated sox effect (filter, replaced by sinc).
Fixed a number of bugs.

New for release 2.5.3:

Added a database layer to some of the code.
Cleaned up some of the code.
Fixed a number of bugs.

New for release 2.5.2:

Fixed a few bugs in version 2.5.1 that did not pointed to the correct path in some scripts.

New for release 2.5.1:

Fixed a few bugs in version 2.5.0 that did not followed the current permission system, could deny access even when set to allow it.

New for release 2.5.0:

Added option for external authentication, in particular using Wordpress.
Added option to always show particular KML layers in map.
Cleaned up some of the code.

New for release 2.4.2:

Fixed a bug in the add files system that parsed incorrectly the time when the file was made.

New for release 2.4.1:

Exclusive use of v3 of the Google Maps API.
Simplified size calculation for exporting files.

New for release 2.4.0:

Added the option to use Google Maps v3. The version 2 has been deprecated by Google.
Added a system for quality control by using technical details of the files, like filesize and duration, to detect problem files. In addition, if R is enabled in the server, additional indices are extracted.
The process to add files to the database can be done in another machine.
Added a license information field, with options for Copyright or CreativeCommons.
Fixed a number of bugs.

New for release 2.3.1:

Moved some menu options to make them easier to use.
Fixed a number of bugs.

New for release 2.3.0:

The system now will use all the resources of the server to draw the spectrogram and waveform images as well as to write the mp3 files when these files do not exist. This feature will take advantage of current multicore computers.

New for release 2.2.1:

Fixed a small bugs that was using an old name for a database field in version 2.2.0.

New for release 2.2.0:

Added an XML data export option. This is a first version, but should help when analyzing data using other languages or systems.
New Pumilio Chorus index of Pumilio instances.
Added quality control flags.
Improved way to document the sensors used.
Fixed small bugs and made improvements in the code.

New for release 2.1.2:

Added site selection to the main menu.
New script to convert flac files to wav files in the server (useful for analysis nodes).
Fixed small bugs and made improvements in the code.

New for release 2.1.1:

New HTML5 file uploader (plupload) that allows to easily upload many files and avoids the problems with PHP size limits.
Added form to delete multiple files at the same time in gallery view (admin users only).
Updated some libraries, like JQuery.
Better advanced search by using date pickers and sliders.
Notes for KML layers now appear as a JQuery UI dialog.
Fixed small bugs and made improvements in the code.

New for release 2.1.0:

Changed the way the files are organized. This was necessary to hold large archives with thousands of files and avoid problems of too many files in a directory.
Fixed small bugs and made improvements in the code.

New for release 2.0.3:

Added option to edit the metadata of all files in a collection or site.
Temporary fix for Chrome when browsing summary view of files. Now tries first to play audio with Flash, HTML5 player was not working with multiple files.
Fixed small bugs and made improvements in the code.

New for release 2.0.2:

Replaced the JWPlayer with an older version to avoid a conflict with the analysis window.
Added a maintenance mode (just add an empty file called 'maintenance' to the root of the application).
Added option to delete a whole collection.
Fixed code to display marks in mono and stereo files correctly.
Added a default option for KML/KMZ layers.
Fixed a bug in IE where pulldown menus did not work.
Fixed small bugs and made improvements in the code.

New for release 2.0.1:

Updated the R plugin for v. 2.x.
Fixed bug when the spectrogram palette has not been selected.

New for release 2.0.0:

New sound player that uses the HTML5 audio for newer browsers or Adobe Flash for versions that do not support HTML5 yet.
Renamed the "sources" to "collections" as mayor organizational units. The database was changed too. This means that scripts and tools written for version 1.x will need to be updated for version 2.x. Helper scripts were updated to work with the new version.
Users can now change the theme, which is stored in a cookie.
Fixed many bugs and made improvements in the code.

New for release 1.7.0:

Added a new feature: allowing admin users to specify KML/KMZ layers to have as options for the maps.
Small improvements to the visualization window.
Fixed small bugs and made small improvements.

New for release 1.6.2:

Added an option to display the marks, or annotations, on a file by default.
Fixed bug on the addmarks script that prevented it from working.
Fixed other small bugs.

New for release 1.6.1:

Fixed small bug where it shows the sql query in advanced search results.

New for release 1.6.0:

Added a search tool for Sound ID or by characteristics like date, time, duration, site, sampling rate, and number of channels.
Fixed other small bugs.

New for release 1.5.3:

Fixed bugs in the administrative options that prevented some features.

New for release 1.5.2 (removed due to bug in administration options):

Fixed bugs in the administrative options.
Updated the process and check_auxfiles scripts.
Fixed other small bugs.

New for release 1.5.1 (removed due to serious bug):

All spectrograms will show the maximum acoustic frequency that it displays.
An option for the maximum acoustic frequency for the spectrograms was added to the administration menu.
The layout of the administration menu and browsing pages were improved.
Fixed other small bugs.

New for release 1.5.0:

New option to display the recordings side-by-side from up to three sites.
Fixed other small bugs.

New for release 1.4.1:

Easier way to import files from Wildlife Acoustics SongMeter boxes. Date and time are automatically determined from the filename.
Added an option to find and delete orphaned files. This should be a rarely used option.
Fixed other small bugs.

New for release 1.4.0:

Cleaned up some of the code, moved some administrative options to the admin panel instead of the config.php file.
Easier management of users.
Added an easy way to import files from the field, for example, when using Wildlife Acoustic recorders. * The only data you need to provide is the site, the rest is determined from the file name.
Replaced a JS plugin with a standard tag.
Added a file export option. You can export all the files in a source or site to a tar or zip file with a csv file that contains the data of each file.
Fixed other small bugs.

New for release 1.3.1:

This version was removed due to serious bugs.
New for release 1.3.0:
Added a script job management system. Now you can schedule jobs on sample sets to be run in other computers.
Fixed other small bugs.

New for release 1.2.3:

Fixed a bug in the map browsing page. If the browser does not have JavaScript enabled, it will say so.
Fixed other small bugs when JS is not enabled (which is not recommended) and styling.

New for release 1.2.2:

Added a management window for the marks in the files.
Fixed a bug in the version checking code.
Fixed other small bugs.

New for release 1.2.1:

Added a large viewing player when viewing the file details.
Added the option for optional plugins. A sample one available in the plugin folder will run an R script on all the files of a source.
Added an option to create a sample set of all the sources based on percentages.
Small performance improvements to the code and the database.
Fixed several bugs and other minor errors.

New for release 1.1:

The administration area indicates when a new version is available.
Added a maintenance option to check for missing data and to optimize the database.
Since this software is intended for scientific use, a feature was added that allows to create sample sets from the sound files in the archive.
Preview sound files (mp3) of files with a sampling rate of less than 44.1kHz should be smaller. This will also improve speed of the application.
Small performance improvements to the code and the database.
Fixed several bugs and other minor errors.

New for release 1.01:

The sound archive can be browsed using a gallery of the spectrograms.
An administrative option allows to create all the preview sounds and images for the files.
Added an easy way to upgrade the database structure.
Small performance improvements to the code and the database.
Fixed several bugs and other minor errors.

New for release 1.0:

Added a tagging system and a tag cloud to the homepage.
The sound preview now displays the whole range of frequencies of the sound, up to the Nyquist frequency of the file.
Added a progress bar when importing files from another database or spreadsheet.
Small performance improvements.
Fixed several bugs and other minor errors.

New for release 0.4.3:

Added a GoogleMaps browsing option. Sounds can be selected by site and date using a GoogleMaps application.
Browsing the sounds can now play a sound preview in mp3 format as well as display the spectrogram and waveform.
User passwords can be recovered using email, if the server supports sending email.
Fixed several bugs and other minor errors.

New for release 0.4.2:

Added a MySQL-backed sound archive. Sounds can be uploaded one by one or imported from another database or a spreadsheet using comma-separated values.
Added the JQuery UI Start theme for forms.
Fixed several bugs and other minor errors.
Now requires at least version 14.3.0 of SoX (included with Ubuntu 9.10).

New for release 0.3.2:

Fixed several bugs and other minor errors.
Sound file information is now parsed using sox or flac instead of audiolab.
The list of file formats available is huge, depending on the local sox installation. My development machine (Ubuntu 9.10 with SoX v14.3.0) lists the following options: 8svx aif aifc aiff aiffc al amb au avi avr caf cdda cdr cvs cvsd cvu dat dvms f32 f4 f64 f8 fap ffmpeg flac fssd gsm hcom htk ima ircam la lpc lpc10 lu m4a mat mat4 mat5 maud mp2 mp3 mp4 mpg nist ogg paf prc pvf raw s1 s16 s2 s24 s3 s32 s4 s8 sb sd2 sds sf sl smp snd sndfile sndr sndt sou sox sph sw txw u1 u16 u2 u24 u3 u32 u4 u8 ub ul uw vms voc vorbis vox w64 wav wavpcm wmv wv wve xa xi

New for release 0.3.1:

Fixed several bugs and other minor errors.
The default media encoder is now LAME for mp3 files.
Added an option to obtain files from other sound databases or from the web.

Pumilio is maintained by ljvillanueva