10th Cracow Cognitive Science Conference

12-13 May 2018, Jagiellonian University, Kraków, Poland








Sensory substitution (SS) involves transposition of the characteristics of one sensory modality into stimuli of another sensory modality using sensory substitution devices (SSD). We propose workshops on Sensory Substitution Devices, where we would compare three different SSDs: ‘The vOICe’, ‘EyeMusic’ and ‘The Colorophone’.

Each device has a different algorithm which slightly alters limits of usage of each of them. ‘The vOICe’ and ‘EyeMusic’ use sweep-line technique, which produces so called ‘soundscape’ - the composition of sounds representing perceived image as a whole but doesn’t work in real time fashion. Soundscapes map the x-axis of captured images to the time domain and the y-axis to the frequency domain. The bottom to top pixels of a pixel column are assigned frequency ranges from low (bottom) to high (top). In ‘The vOICe’ the columns are scanned across the image and brightness is translated into volume. In ‘EyeMusic’ the algorithm is also based on the image sweep- line technique but it is different from the vOICe - the sounds underlying the generated soundscapes in algorithm are recordings of musical instruments, where each instrument represents a different color.

‘The Colorophone’ device, on the other hand, enables continuous conversion of information about color into sound based on algorithm which makes use of RGBY colour matrix values: each of the four primary colours is assigned to a different sound coupled with sound frequency. The coupling system makes it possible to manipulate the sound frequencies and amplitudes. All other colours are represented by a mix of these four primary colours according to the principles of the RGBY matrix. RGBY values (0-255) are coupled with sound amplitudes so that recognition of a colour requires the identification of one or two sound frequencies (tones) and their amplitudes (volume) and a noise (indicating lightness or white color).

During workshop we will introduce this three SSDs and sensory substitution phenomena and then we intend to divide participants into groups, each of which will be working on similar tasks preceded by a 15 minutes training. After every of such tasks we will discuss limits, comfortability of usage and accuracy of each device by comparing them.

The workshop will be prepared and delivered by:
Paweł Gwiaździński, Patrycja Bizoń, Katarzyna Hat, Magdalena Reuter.

This workshop is part of project OPUS 12, ID: 352990, OPUS, HS6, titled „Cognitive and neural plasticity and the subjective experience. Interdisciplinary analysis of sensory substitution”, financed by Polish National Science Center (NCN) and directed by dr hab. Michał Wierzchoń



Institute of Psychology
Ingardena 6, Cracow, Poland