TAMILNADU STATE COUNCIL FOR SCIENCE AND TECHNOLOGY (Autonomous body of Government of Tamilnadu) DOTE Campus, Chennai – 600 025
PROGRAMME FOR BRIDGING THE GAP IN RESEARCH FUNDING FOR RESEARCH SCHOLARS IN COLLEGES (RFRS)(2018-2019)
1.Name and Institutional Address
of the Research Scholar (With Phone,
Fax, E-mail etc) : K.RATHI
Department of CSE
National Engineering College,
Kovilpatti – 628503.
2. Permanent /Residential address of the
Candidate with phone,fax,email etc : 36/1, Bolden Puram – Third Street,
Thoothukudi – 628003.
[email protected] 8220133306
3. Sex : Female
4. Date of Birth : 05- JUNE-1994
5. Nativity : Thoothukudi
6. Broad area of Research : Engineering & Technology
7. Educational Qualification :
Degree University Subject(s) Grade/Mark Award/ Distinction
B.E., Anna University – Chennai Electronics & Communication Engineering CGPA: 7.61 First Class
M.E., Anna University – Chennai Communication
Systems CGPA: 9.264 First Class with Distinction
8. Broad area of Research and date of Ph.D
Registration : Reconstruct Images from Brain Using
Digital Image Processing
04 – JAN -2018.
9. List of Research papers already published :
Author Title Journal Page No. Impact factor
K.Rathi Multi-Robot For Safeguard Advances in Natural and Applied Sciences,11(5),2017 152-154 10. Brief write up (abstract) of the proposed research work:
The visual cue understandings of the human normally happen at the occipital lobe of the brain with an amazing mapping process. Nowadays, the ability of reconstruction of visual cues from the human brain activities using functional Magnetic Resonance Imaging (fMRI) is an active research field under neuropsychological research. This would bring scope for those suffer on the autism spectrum disorder (ASD) or those affected with psychotic disorders. This capability is also called as theory of mind (ToM). Also, this ability has evolved over time, especially in primates, because of selection pressure brought about by increasing social complexity. This hypothesis is called “social brain hypothesis”. A neuro-physiological hypothesis of social cognition is involved in the following brain regions: the amygdala, orbitofrontal cortex (OFC), and superior temporal gyrus (STS).
Few researchers have suggested mapping techniques that can reconstruct a digital image of what a person is picturing in their mind, simply by considering their electrical activity of the brain as fMRI voxel information. During the gathering of voxel information for experiments, normally it is required to display the participants the MNIST digit images, faces of many individuals, one-by-one, on the computer screen, in parallel to recoding event of their brain activity.
There are open research challenges to develop efficient reconstruction models to predict the latent space of mapping the human vision from voxel data. Essentially, this mapping algorithm identifies the pattern of brain signals related to the mental image that the volunteers were holding in their mind and reproduced it as a digital image.
The main goal of this project proposal is to propose a novel image reconstruction method, in which the voxel data and corresponding pixel information will be correlated with deep learning approach based on latent-variable distributions. This reconstruction method mainly depends on the observed brain activity patterns in the form of physiological modalities fMRI or EEG. In this work, we will also study the scope for exploiting the spatio-temporal EEG information to determine the neural correlates of visual scene representations and to reconstruct the appearance of the corresponding stimuli.
11. Detailed proposal
Defining the research problem
Develop efficient reconstruction models to predict the latent space of mapping the human vision from voxel data, and also reduce the excessive computational complexity of the reconstruction images. This proposed project would bring scope for those suffer on the autism spectrum disorder (ASD) or those affected with psychotic disorders.
Review of Literature
1) The neural dynamics of facial identity processing: Insights from EEG-Based pattern analysis and image reconstruction
Dan Nemrodov, Matthias Niemeier, Ashutosh Patel, and Adrian nestor (2018)
Recorded human electroencephalography (EEG) data associated with viewing face stimuli, then exploit spatiotemporal EEG information to determine the neural correlates of facial identity representations and to reconstruct the appearance of the corresponding stimuli. Further, aggregate data from a larger interval support robust reconstruction results, consistent with the availability of distinct visual information over time. The time course of face processing while, methodologically they demonstrate the feasibility of EEG-based image reconstruction.
2) Sharing deep generative representation for perceived image reconstruction from human brain activity
Changde Du, Changying Du, Huiguang He (2017)
Two main challenges that hinder the development of effective models are the perplexing fMRI measurement noise and the high dimensionality of limited data instances. The reconstruction of visual stimulus as the Bayesian inference of missing view in a multi-view latent variable model.
3) Deep image reconstruction from human brain activity
Guohus Shen, Tomoyasu Horikawa, Kei Majima and Yukiyasu Kamitani (2017)
A novel image reconstruction method, in which the pixel values of an image are optimized to make its DNN features similar to those decoded from human brain activity at multiple layers, and found the generated images, resembled the stimulus images and the subjective visual content during imagery. Furthermore, human judgement of reconstructions suggested the effectiveness of combining multiple DNN layers to enhance the visual quality of generated images. The result suggested that hierarchical visual information in the brain can be effectively combined to reconstruct perceptual and subjective images.
4) Generic decoding of seen and imagined objects using hierarchical visual features
Tomoyasu Horikawa, and yukiyasu Kamitani (2017)
Brain decoding through machine learning analysis of functional magnetic resonance imaging activity has enabled the interpretation of mental contents, including what people see, remember, imagine, and dream. First demonstrate that visual feature values of seen objects calculated by the computational models can be predicted from multiple brain areas, showing tight associations between hierarchical visual cortical areas and the complexity levels of visual features. In addition, the stimulus-trained decoders can be used to decode visual features of imagined objects, providing evidence for the progressive recruitment of hierarchical neural representations in a top-to-bottom manner. The features predicted from brain activity patterns are useful for identifying seen and imagined objects for arbitrary categories.
5) Feature-based face representations and image reconstruction from behavioral and neural data
Adrian Nestor, David C.Plaut, and marlene Behrmann (2015)
The reconstruction of images from neural data can provide a unique window into the content of human perceptual representations. Using twofold task of deriving features directly from empirical data and of using these features for facial image reconstruction. To map cortical areas that exhibit separable patterns of activation to different facial identities and then construct confusability matrices from behavioral and neural data in these areas to determine the general organization of face space. To investigate a range of facial properties, the broad organization of behavioral face space reflects that of its neural homolog, and high-level face representations retain sufficient detail to support reconstructing the visual appearance of different facial identities from either neural or behavioral data.
6) Linear reconstruction of perceived images from human brain activity
Sanne Schoenmakers, Markus Barth, Tom Heskes, Marcel Van Gerven(2013)
A straightforward linear Gaussian approach, where decoding relies on the inversion of properly regularized encoding models, which can still be solved analytically. In order to test, functional magnetic resonance imaging data under a rapid event related design in which subject were presented with handwritten characters.
7) Decoding representations of face identity that are tolerant to rotation
Stefano Anzellotti, Scott L. Fairhall, and Alfonso Caramazza(2013)
Investigated the representation of individual faces in the brain, but it remains unclear whether the human brain regions that were found encode representations of individual images or face identity. Use multi-voxel pattern analysis in the human ventral stream to investigate the representation of face identity across rotations in depth, a kind of transformation in which no point in the face image remains unchanged.
8) Decoding patterns of human brain activity
Frank Tong and Michael S. Pratte (2012)
Information can be decoded from noninvasive measures of human brain activity. Analyses of brain activity patterns can reveal what a person is seeing, perceiving, attending to, or remembering. Moreover, multidimensional models can be used to investigate how the brain encodes complex visual scenes or abstract semantic information. Such feats of “brain reading” or “mind reading,” though impressive, raise important conceptual, methodological, and ethical issues.
9) Encoding and decoding in fMRI
Thomas Neselaris, Kendrick N.kay, Shinji Nishimoto, Jack L.Gallant (2011)
A more recent development is the voxel-based encoding model, which describes the information about the stimulus or task that is represented in the activity of single voxels. Encoding and decoding are complementary operations: encoding uses stimuli to predict activity while decoding uses activity to predict information about the stimuli.
10) Visual image reconstruction from human brain activity using a combination of multiscale local image decoders.
Yoichi Miyawaki, Hajime Uchida, Okito Yamashita, Masa-aki Sato, Yusuke Morito, Hiroki C Tanabe, NorihiroSadato, and YukiyasuKamitani.(2008).
Constraint free visual image reconstruction is more challenging, as it is impractical to specify brain activity for all possible images. Reconstruct the visual images by combining local image bases of multiple scales, whose contrasts were independently decoded from fMRI activity by automatically selecting relevant voxels and exploiting their correlated patterns. Reconstructed the lower order information such as binary contrast patterns using a combination of multi-scale local image bases whose shapes are predefined. This approach provides an effective means to read out complex perceptual states from brain activity while discovering information representations in multi-voxel patterns.
To assimilate the human vision process related to brain activities.
To visualize perceptual content from the recorded brain activity in the form of voxel / EEG to pixel mapping.
To build Deep Learning Network model for image reconstruction with brain fMRI image sets.
To reduce high computational complexity (due to highly-correlated covariance computation) using parallel algorithms.
Methodology of research
The layered architecture for the proposed methodology is shown below:
Handle these experiments on two public fMRI datasets. Dataset1 contains a hundred handwritten gray-scale digits(equal number of 6s and 9s) at a 28*28 pixel resolution taken from the training set of MNIST database and the fMRI data from visual areas(V1,V2,V3). Dataset2 contains 360 gray-scale handwritten characters (B, R, A, I, N, S) at a 56*56 pixel resolution and the fMRI data of V1, V2 taken from three subjects. For each character, 60 individual instances were centrally presented during the experiment. The summarized details of both the dataset is given below:
Datasets No.of Instances No. of pixels No. of voxels No. of ROIs No. of Training
Dataset1 100 784 3092 V1,V2,V3 90
Dataset2 360 784 2420 V1,V2 330
To keep members sharpness they were formally asked for center-on the obsession point and to respond with a catch squeeze when the obsession point changed shading. The obsession point changed shading once every six stimuli on average. Changes were introduced aimlessly yet equally spread over the length of the examination. The analysis went on for 50 min with a self-managed rest period in the center. Later the investigation, a supplementary output was made.
The functional images were gathered with a Siemens Trio 3 T MRI framework with an EPI arrangement utilizing a 32 channel head coil (TR=1.74s, TE=30s, GRAPPA acceleration factor 3, 83? flip angle, 30 slices in ascending order, voxel size 2 × 2 × 2 mm). Head development was confined with foam cushions and a tight portion of tape over the forehead. After useful imaging, an auxiliary output was obtained. In a different session, the functional localizer information was gained, again utilizing an EPI grouping (TR=2s, TE-30s, voxel size=1 × 1 × 1 mm, 192 sagittal slices, FOV=256mm). During acquisition, an eye tracker was utilized to confirm if members were focusing their look.
With the utilization of SPM8 programming, the functional volumes were remade, realigned to the principal sweep of the session and cut time revised. Members moved under 0.5 mm across the sessions. For every special image, which was exhibited twice to the subject, the response of each voxel to the image was processed utilizing a linear model.
Design of Recurrent Neural Network
A Recurrent Neural Network (RNN) is a class of Artificial Neural Network where associations between nodes frame a coordinated diagram along a grouping. The RNNs is to make use of sequential information. In a traditional neural network, all inputs and outputs are independent of each other. A typical RNN looks like:
Step 1: A single time step of the input (xt) is supplied to the network.
Step2: Calculate its current state (ht) using a combination of the current input and the
ht = f (ht-1 ,xt)
Step 3: The activation function is tanh, the weight at the recurrent neuron is whhand the
weight at the input neuron is wxh ht = tanh (whhht-1 + wxhxt )
Step 4: Once all the time steps are completed the final current state is used to calculate the
yt = whyht Step 5: The output (yt) is then compared to the actual output (? t) and the error is generated.
Et (?t,yt) = -?t log (yt)
E (?,y) = – ?t log(yt)
1.Using GPU, the deep learning algorithms work faster .
2.Parallelism can help with better performance.
IBM’s Minsky Server:
Minsky servers are two?socket POWER processor servers with NVIDIA GPU accelerators. Two differentiating Technologies in these POWER processor servers are CPU?to?GPU NVLINK connections, plus Coherent Accelerator Processor Interface (CAPI). NVLink technology is a high?speed processor interconnect that can be used to connect GPUs to CPUs or other GPUs. NVLink eliminates the PCI bottleneck, providing 5x or greater performance than PCIe. NVLink enables the logical integration of multiple GPUs and of CPU and GPU cores. Using these connections, each GPU has direct paged access to both the memory of the host processor and the memory on the sibling GPU. This revolutionary model in GPU computing significantly decreases the programming complexity associated with both basic GPU computing and multi?GPU computing.
Fig: POWER9′ & NVLink
CAPI enables adapters or accelerators plugged into a PCIe slot to access the processor bus using a low latency, high speed protocol interface. CAPI provides a level of cache coherency, plus simplifies and shortens the device path.
Fig: CAPI I/O Model
12. Social relevance and useful of the proposed research
Analyzing the real-time EEG/fMRI channels of information for handling the developmental disabilities. This proposed project would bring scope for those suffer on the autism spectrum disorder (ASD) or those affected with psychotic disorders.
13. Expected outcome of the research work
This project, on completion, will definitely be the best resource to demonstrate the brain functional connectivity. As a means of minimizing the high computational complexity using parallelism algorithm. Also the system will support the enhanced training session for fMRI/EEG neurofeedbacks.
14. Details of Research Guide (Name and
Institutional address along with Phone,
Fax, email etc, brief bio-data of the Guide) : Dr.V.Gomathi
Professor & Head / Dept.of. CSE
National Engineering College (Autonomous)
K.R.Nagar – 628 503.
[email protected] , 94864 49790
Brief bio-data of the Guide : Refer Annexure 1
15. Facilities available at the college for the research work proposed
Facility Research / Sharing Basis
Intel Core i5 Processor, 2013 Dedicated to Research
fMRI Dataset Dedicated to Research
Wi-Fi Shared with Academic
E-Journal & Library Shared with Academic
Place : Kovilpatti
Date : 21 .08.2018 Signature of the Applicant
Signature of the Signature of the Signature of the
Guide Head of the Principal/ Head of the
Department Institution with seal