Blog

Installing a specific version of Bioconductor

2016-11-12T00:00:00+01:00

Bioconductor provide a list of R packages to analyse high-throughput genomic data. Every six months a new version of Bioconductor is released fixing the version of the included packages. It is always nice to come back to the same version of packages that generated the results.

Bioconductor versions are bound to R versions. So depending of the version of R you are using a limited number of Bioconductor versions are available. Using the default installation method of the Bioconductor the last version available for the R version is installed. We will see how to install a different version.

Install a specific version of Bioconductor

To install a specific version of Bioconductor, we need to use the installer of that version. To do so, we need to install the installer of the wondered version. The installer is in the BiocInstaller package we just need to use the correct repository to install it.

For example to install Bioconductor 3.1:

install.packages("BiocInstaller",
                 repos="http://bioconductor.org/packages/3.1/bioc")

Then load the BiocInstaller package as usual with library.

library("BiocInstaller")

Then install Bioconductor packages as usual with then biocLite() function.

Downgrade of already installed packages

In case you already install a more recent version of Bioconductor you can easily downgrade the packages to fit the Bioconductor version.

After loading the BiocInstaller you can have information about the Bioconductor installation with the biocValid().

For example

> biocValid()

* sessionInfo()

R version 3.2.4 Revised (2016-03-16 r70336)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Debian GNU/Linux 8 (jessie)

locale:
 [1] LC_CTYPE=fr_FR.UTF-8       LC_NUMERIC=C
 [3] LC_TIME=fr_FR.UTF-8        LC_COLLATE=fr_FR.UTF-8
 [5] LC_MONETARY=fr_FR.UTF-8    LC_MESSAGES=fr_FR.UTF-8
 [7] LC_PAPER=fr_FR.UTF-8       LC_NAME=C
 [9] LC_ADDRESS=C               LC_TELEPHONE=C
[11] LC_MEASUREMENT=fr_FR.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

other attached packages:
[1] BiocInstaller_1.18.5

loaded via a namespace (and not attached):
[1] tools_3.2.4

* Packages too new for Bioconductor version '3.1'

                  Version  LibPath
annotate          "1.48.0" "/home/tblein/bin/R_3.2.4/lib/R/library"
AnnotationDbi     "1.32.3" "/home/tblein/bin/R_3.2.4/lib/R/library"
Biobase           "2.30.0" "/home/tblein/bin/R_3.2.4/lib/R/library"
BiocGenerics      "0.16.1" "/home/tblein/bin/R_3.2.4/lib/R/library"
BiocParallel      "1.4.3"  "/home/tblein/bin/R_3.2.4/lib/R/library"
Biostrings        "2.38.4" "/home/tblein/bin/R_3.2.4/lib/R/library"
DESeq2            "1.10.1" "/home/tblein/bin/R_3.2.4/lib/R/library"
genefilter        "1.52.1" "/home/tblein/bin/R_3.2.4/lib/R/library"
geneplotter       "1.48.0" "/home/tblein/bin/R_3.2.4/lib/R/library"
GenomeInfoDb      "1.6.3"  "/home/tblein/bin/R_3.2.4/lib/R/library"
GenomicAlignments "1.6.3"  "/home/tblein/bin/R_3.2.4/lib/R/library"
GenomicRanges     "1.22.4" "/home/tblein/bin/R_3.2.4/lib/R/library"
IRanges           "2.4.8"  "/home/tblein/bin/R_3.2.4/lib/R/library"
Rsamtools         "1.22.0" "/home/tblein/bin/R_3.2.4/lib/R/library"
rtracklayer       "1.30.4" "/home/tblein/bin/R_3.2.4/lib/R/library"
S4Vectors         "0.8.11" "/home/tblein/bin/R_3.2.4/lib/R/library"
XVector           "0.10.0" "/home/tblein/bin/R_3.2.4/lib/R/library"
zlibbioc          "1.16.0" "/home/tblein/bin/R_3.2.4/lib/R/library"

downgrade with biocLite(c("annotate", "AnnotationDbi", "Biobase", "BiocGenerics",
  "BiocParallel", "Biostrings", "DESeq2", "genefilter", "geneplotter",
  "GenomeInfoDb", "GenomicAlignments", "GenomicRanges", "IRanges", "Rsamtools",
  "rtracklayer", "S4Vectors", "XVector", "zlibbioc"))

Erreur : 18 package(s) too new

So as indicated by biocValid(), to downgrade all the package you just need to run the biocLite() function with the packages you would like to downgrade.

Sources

How to organise the source code of "templates"

2016-07-23T00:00:00+02:00

In C and C++, we normally separate the declaration and the definition of a function in two separated files: the header file contain the declaration of the function (.h or .hh file) and the source code itself which contain the definition of the function (.c or .cpp file). When we would like to do the same with function containing template we get an error of type undefined reference to when the function is first used in code.

An example

For example let's suppose that the addition function return the sum of two numbers of same type.

First the header file (addition.hh):

template<typename Type> Type addition(Type a, Type b);

Then the function source code (addition.cc):

template<typename Type> Type addition(Type a, Type b)
{
    return a + b;
}

Finally a small program using this function (main.cc):

#include "addition.hh"
void main()
{
    int a = 3;
    int b = 2;
    int c = 0;
    c = addition(a, b)
}

While the code looks correct, the links creation will broke with the following error:

/tmp/ccEpROXj.o(.text+0x17c): In function `main':
: undefined reference to `addition(int, int)'

Why?

There are different reasons that this error appear. It is mainly because a template is not a function but a model used to generate the function. When it used by a program to generate the real function, the definition should be known and not only it's declaration.

How to solved it.

There is several solution to that problem.

The insertion of the definition in the header file.

The first one is to include the definition in the header file. For that, we only need to fuse the two files (header and source code) in only one file or to include the source code file with an include preprocessing command at the end of the header file. This solution is the easiest one, but not the results is not so clean. Indeed, some compilers can increase dramatically the size of the resulting binary.

Adding a template specialisation

FIXME: To be confirmed

We can also add the declaration of the specialisation that we want in the source code of the function.

For example, for the previous addition function, we can specify the declaration for types int, float and double:

template<typename Type>
Type addition(Type a, Type b)
{
    return a + b;
}

template int addition<int>(int, int);
template float addition<float>(float, float);
template double addition<double>(double, double);

By using the `export` keyword.

FIXME

Sources

abort: requirement 'fncache' not supported!

2016-07-23T00:00:00+02:00

It can append that when we try to access a Mercurial repository we got the following error:

abort: requirement 'fncache' not supported!

It append when a "too old" version of Mercurial is used. Indeed, from version 1.1, Mercurial used a new repository format (fncache), which is no more readable by the former versions. The reverse is not true, and newer version are able to read old repositories.

To solve the problem we can use a version of Mercurial able to read the two type of repository and convert it to the old format using the following command:

hg --config format.usefncache=0 clone --pull A B

With A the initial repository and B the converted repository.

To completely deactivate the new repository format, a modification the configuration file of Mercurial is needed (~/.hgrc):

[format]
usefncache = False

Sources

fncacheRepoFormat

Unrealated repository fusion (mercurial abort: repository is unrelated)

2016-07-23T00:00:00+02:00

When we try to include a repository in an other while they never had any common point we got the following error:

$ hg push ../second
mercurial abort: repository is unrelated

To go above it we need to force the fusion with the -f option:

$ hg push -f ../second

Sources

hg man page

Deconnected subversion with SVK

2016-07-23T00:00:00+02:00

SVK installation

apt-get install svk

Create a local repository with the following command:

svk depotmap --init

Create a local repository mirror

svk mirror svn://adresse_du_depot_distant.ex //local/nom_local_du_depot
svk sync //local/nom_local_du_depot

Create a workingcopy from the local repository mirror:

svk checkout //local/nom_local_du_depot repertoire

Local use of SVK

Add some files/folders to version control

svk add fichier

Commit to the loacl repository mirror

svk commit

Push the modifications of the local mirror repository to the distant repository:

svk push

Sources

SVK par les Mongueurs de Perl

po regex

2012-09-04T00:42:19+02:00

Some notes to be able to construct a small program able to apply a regex to translated string of a po file

Will be done certainly in python:

polib python module to be able to handle the po files seems quite simple to use according to the Quick start guide. In addition polib is included in Debian repository.
Finally the python regex module re. The official Howto on regex in python. The regex are not set as in perl but with a regex, the on the perl regex the two parts could be extract easily.
command line arguments parse with argparse as usual

Redirecting sderr to a pipe

2012-07-31T11:36:06+02:00

Programs are sending message on two different steam of data: standard output and standard error. Standard output, notes as stdout, is where the program is writing the output data. Standard error or stderr is to output the error messages or diagnostics. With this two stream we can therefore separate between these two kind of output and process them separately. A third stream of data exist, that is standard input or stdin, which correspond to the entry of data for a program.

Classical shell programming involved to play with these different stream of data, where the stdout of one program is pipe to the stdin of an other program to get the desired functionality. One of this task is to filter the output of a command with grep.

$  ls -1
Desktop
Documents
Pictures
Videos
Download
Music

$  ls -1 | grep D
Desktop
Documents
Download

However, pipe is only for redirection of stdout to the stdin. Sometime we would like to filter the stderr and not the sdtout. Therefore we need to redirect the stream so they can be processed correcrtly.

First we can redirect stderr to stdout and mix the two output that will be then filtered:

$ myprog 2>&1 | grep something

In the same time we can destroy the output of stdout and only keep stderr to be filtered:

$ myprog 2>&1 > /dev/null | grep something

You have to be carefull in the order of the command since the following do not seems to work as expected, both stream being redirected to /dev/null

$ myprog > /dev/null  2>&1| grep something

Source

How can I redirect stderr to a pipe?

po files handling

2012-07-05T22:41:31+02:00

po files are used by programs to translate the messages in different languages. Thus they contain a list of all translatable strings extracted from the code and their corresponding translation, in a given language. Here are given some information to manipulate and few resources to help the translation, mainly derived from experiments in for Debian project translation to French.

File formating

For ease of reading text files on any type of screen it is common to limit the size of their line to 80 characters. The following command allows to cut the lines at 80 characters while maintaining the specific labeling of po files:

$ msgcat fr.po -o fr.po

You can specify the length of a line to another value if necessary with the -w option. So to format a po file with 40 characters:

$ msgcat -w40 fr.po -o fr.po

Update of a file while already start the translation

When translating sometimes the original po file have been updated while your translation is not finished. The following command can update your po file (fr.po) with new data from the new file (fr.new.po):

$ msgmerge --update --previous fr.po fr.new.po

You can continue on your translation on the updated fr.po file.

Resources

General

Translation proposition database based on already translated FLOSS projects: http://open-tran.eu/

Debian specific (French translation team)

Help

Status and coordination

Perl and UTF-8

2012-05-08T01:20:03+02:00

Like with python, the handling of [[!wikipedia UTF-8]] string in perl is not straight forward since this language was not originally design to handle Unicode strings of characters.

Correct length of UTF-8 string

[[!wikipedia ASCII]] characters are coded on 8bits that is on one octet. That explain the limited number of different characters and the different encodings according to language. Unicode try to solve it by encoding it on several a dynamic number of octet depending of the characters. Therefore some characters are only coded on one octet (the one that are present in the [[!wikipedia ASCII]] set) some other on two octets.

By default in perl the strings are [[!wikipedia ASCII]] and therefore to determine their length with the length function it only count the number of octet. That fit perfectly when the characters are coded only on one octet but in a lot of language the some used characters are coded on several octets. The length is therefore longer, for example in the following script:

#! /usr/bin/env perl
my $string = "Ceci est une chaîne avec trois caractères codés sur deux octets";
print length($string) . "\n";

$ ./test.pl
66

In fact there are 63 characters in this string.

To get the real string length a conversion of the format is needed. For that the Encode module need to be use to decode the [[!wikipedia UTF-8]] string before measuring its length:

#! /usr/bin/env perl
use Encode;
my $string = "Ceci est une chaîne avec trois caractères codés sur deux octets";
print length(Encode::decode("utf-8", "$string")) . "\n";

$ ./test.pl
63

This time the correct size of the string is returned

Wide character in print

When printing some [[!wikipedia UTF-8]] string on the console it may append that the following error appear:

Wide character in print at test.pl line 3.

To remove of it just add the folling entry at the beginning of your script:

use encoding "utf-8";

Resources

Wide character in print… warning in Perl

Preventing multiple executions of a Makefile

2012-05-01T17:21:48+02:00

Often the automatic builds are controlled by the make tools using the rules defined in so named Makefile. make is very nice since it allow a smooth control of the build with a direct link with dependencies. Therefore only what is need to be build is built. A rule is executed only when the dependencies are younger than the target for that rule.

To automate the builds regularly the construction might be run regularly via cron jobs. However, if the time in between the make runs is smaller than the time required to perform a particular task, this task will be run several time and filled the resources of the computer. To prevent it we can setup a protection on the Makefile to prevent it to be executed several times at a given time.

Lock file

To prevent the multiple execution of the Makefile we will use a so called lock file. When the Makefile is execute it will create a file with a particular name. This is our lock file. When it finish to execute, it just delete this file. In case the Makefile is run a second time in parallel, it will see that the lock file exist and stop the processing.

Setting up lock files inside a Makefile

To determine the existence of the file we will use the unix command ls that will return the name of the lock file if it exist else nothing. We will catch it in a make variable:

LOCKFILEEXIST = $(shell ls $(LOCKFILE) 2> /dev/null)

Then we test if this variable is empty or not. In case it is not empty, we exit with an error:

ifneq ($(LOCKFILEEXIST), )
$(error "already running")
endif

In case we the lock file does not exist we continue the process and set the lock file.

$(shell touch $(LOCKFILE))

Then follow the rest of the Makefile like usual. We just need to remove the lock file at the end of the rpocessing. For this we add it at the end of the all rules that is the main run.

all:
    @echo "processed"
    $(shell rm -f $(LOCKFILE))

This command should be add to all rules that terminate the run, otherwise future run might be blocked even no Makefile is running

Testing it

The final test Makefile:

# Definition of the lockfile
LOCKFILE = "test.lock"

# Search for the lock file
LOCKFILEEXIST = $(shell ls $(LOCKFILE) 2> /dev/null)

# if the lock file exist exit
ifneq ($(LOCKFILEEXIST), )
$(error "already running")
endif

# Create the lock file
$(shell touch $(LOCKFILE))

# process and remove the lock file
all:
    @echo "processed"
    $(shell rm -f $(LOCKFILE))

Without any lock file the execution is done and like this several time:

$ make all
processed
$ make all
processed

When a lock file exist the Makefile stop with an error:

$ touch test.lock
$ make all
 Makefile:5: *** "already running". Stop.

Writting UTF-8 strings in python

2012-05-01T14:48:48+02:00

When processing some strings in python you may have to deal with special characters. You test your code with some output on the standard output and everything is working. However when you want to write your string you have complains like the following:

UnicodeError: ascii codec can't decode byte 0x85 in position 255: oridinal not in range(128)

What is the problem?

The problem comes from the fact that when writing in the file, python use the [[!wikipedia ASCII]] coding that code characters on 8bits and have only 128 different characters, while your string is encodes with a higher number of characters. On most current Unix the default character encoding is [[!wikipedia UTF-8]] that code characters with a variable number of bits. So it may appends that some of the characters you want to write are no more in the set of [[!wikipedia ASCII]]. There is no problem when printing on the standard output, because your console is certainly in [[!wikipedia UTF-8]] also.

How to solve it?

In python there are to sort of structures that handle list of characters: strings and unicode. The strings are using [[!wikipedia ASCII]] and unicode are using Unicode as their name shows it.

To be able to write a string encodes with [[!wikipedia UTF-8]] you first need to transform it as unicode.

> my_utf8_string = "A string with some UTF-8 characters °"
> my_utf8_string
'A string with some UTF-8 characters \xc2\xb0'
> converted_string = unicode(my_utf8_string, "utf-8")
> converted_string
u'A string with some UTF-8 characters \xb0'

Then you need to open the file as [[!wikipedia UTF-8]] with the codecs python module and save your unicode string as with a normal file handling.

import codecs
with codecs.open("/tmp/test", "w", encoding="utf-8") as outFile:
    outFile.write(converted_string)

That's all.

Resources

C++ links

2010-09-09T00:00:00+02:00

Blog

Installing a specific version of Bioconductor

Install a specific version of Bioconductor

Downgrade of already installed packages

Sources

How to organise the source code of "templates"

An example

Why?

How to solved it.

The insertion of the definition in the header file.

Adding a template specialisation

By using the export keyword.

Sources

abort: requirement 'fncache' not supported!

Sources

Unrealated repository fusion (mercurial abort: repository is unrelated)

Sources

Deconnected subversion with SVK

SVK installation

Create a local repository mirror

Local use of SVK

Sources

po regex

Redirecting sderr to a pipe

Source

po files handling

File formating

Update of a file while already start the translation

Resources

General

Debian specific (French translation team)

Help

Status and coordination

Perl and UTF-8

Correct length of UTF-8 string

Wide character in print

Resources

Preventing multiple executions of a Makefile

Lock file

Setting up lock files inside a Makefile

Testing it

Writting UTF-8 strings in python

What is the problem?

How to solve it?

Resources

C++ links

By using the `export` keyword.