EEG representing on brain surface using volume rendering

1—Brain research is challenging. One of the standard research methods is electroencephalography (EEG). As a rule, this study is presented in the form of graphs. This article describes an approach in which this data is mapped onto a brain model generated from an magnetic resonance imaging (MRI) scan. This allows you to look at the EEG study from a different point of view. An MRI scan will also allow you to take into account some of the features of the brain. This is an advantage over mapping just to a brain template. This noninvasive system can be implemented to monitor the patient in real time, for example, during space flight. Keywords—medical IT; magnetic resonance imaging (MRI); electroencephalogram (EEG); volume rendering; gradient calculation; medical image processing; rendering; brain.


INTRODUCTION
Nowadays, there is a set of medical methods that can improve the quality and duration of life. One of the most modern and promising techniques is a broad direction aimed at the artificial growth of organs [1] that can be transplanted to humans. This approach solves many problems at once: 1. The risk of unsuccessful organ transplantation to a person decreases because there is an opportunity to grow organs for a specific person based on his stem cells [2].
2. The problem of the lack of organs for transplantation is being solved.
However, the brain is a special organ because it cannot be transplanted. This presents scientists and medical professionals with the challenge of preventing and treating brain disease. All of this requires a deep understanding of how the brain works. As a rule, such diseases are associated 1 Copyright © 2021 by ESS Journal with neuronal degeneration, for example, their demyelination [3]. This poses the purpose of introducing new methods for studying the behavior of neurons.
A long-standing goal of researchers has been to map the electrical activity of neurons in the brain in an awake, active person. Electroencephalography is one such technique, examining electrical activity on the surface of the cerebral cortex. It is non-invasive and allows research to be carried out while awake and active, because does not require a rigidly fixed position of the patient. As part of a traditional electroencephalographic study, the current method allows the electrical activity of the brain to be displayed on its volumetric model. This method can be important from the point of view of studying the behavior of the propagation of waves of activity over the surface of the brain in the context of various defects: gross disorders, degradation of convolutions (a decrease in their severity), local damage due to hypertension, etc. Potentially, a local reduction of blood flow in the cerebral arteries can be diagnosed, since blood is a supplier of glucose consumed by neurons during its working cycle, and in the case of its lack, the activity of neurons should decrease [4], which can probably be displayed by the proposed method. Usually, such a reduction is detected after conducting magnetic resonance imaging, but the latter method is costly and not always available and requires sophisticated equipment. The electroencephalograph is a simpler and easier-to-transport device. The diagnosis of blood flow reduction is an important diagnosis because it can result from narrowing of the artery, which increases the chance of thrombosis.
In brain research work, as a rule, to show the signal of an electroencephalograph or its interpretation, they use graphs or display the result on a template two-dimensional model of the head, shown above [5,6]. Charts do not ~ 11 ~ provide spatial representation of data, and the template does not allow to take into account the individual characteristics of a particular brain, although it facilitates data averaging. The use of a three-dimensional model of the brain obtained from an MRI image allows a more accurate display of the result for each specific individual.
The goal of this work is to extract voxels corresponding to the brain and display EEG data on it, given the spatial location of the sensors on the surface of the head relative to the surface of the brain.

II. RESEARCH QUESTIONS AND OBJECTIVES
MRI data is a three-dimensional texture, each voxel of which stores a radio frequency response from an elementary volume around a point. Such information is not an optical property; therefore, working with an MRI texture requires calculating various characteristics, as well as mapping the voxel space into the optical property space in one way or another, i.e. informing each voxel about its color, transparency and shading depending on its scalar value, its absolute or relative position.
It should also be taken into account that MRI textures often have different dimensions along one of the axes, which can complicate the visual representation of the processed data. Isolation of the brain is challenging due to the great variability in the quality of MRI data. You need to try different methods that will be associated with the render method.
The article outlined the features of various methods of MRI rendering and brain isolation, which were encountered in the course of work. This is an integral part of it, since, depending on the available data and the desired result, different approaches can have a dynamic assessment of the effectiveness of the application. Thus, the two-dimensional transfer function described below can have a relatively increased accuracy in the brain of animals or a brain that has topological features. Specifically, we address the following research question: How exactly should the MRI texture be processed and what methods should be used to isolate the brain, as well as what methods of working with the EEG should be used to display the EEG data of high quality?

III. PROPOSED SOLUTION APPROACH
Our approach uses an EEG device connected through a set of sensors to the surface of the head and broadcasting a stream of data about the electrical activity recorded by each sensor. The brain is rendered using GPUs and compute shaders, which allows real-time data processing due to high performance on a specific task. The rendering algorithm will be based on the Direct Volume Rendering method. Different methods of brain extraction will be considered, using LOOK-UP-TABLE and one of the automatic algorithms.

IV. MRI TEXTURE RENDERING METHODS
There are many methods for rendering MRI textures. We will highlight two in this paper: 1. Direct volume rendering method 2. Tissue surface rendering method

A. Direct Volume Rendering method
In this MRI technique, the texture is scanned by rays. Each ray is emitted from a screen pixel (output texture), where the screen is a rectangle in an arbitrary region of space, and its pixels are represented by a set of points in this rectangle. The direction of the ray depends on the type of camera used for rendering. In the case of orthography, the rays are directed straight. In the case of perspective -at a variable angle. The MRI texture is a unit cube. In the cycle, the ray makes a constant step from its origin in its direction until it leaves the unit cube of the MRI texture. Each step it samples data from the MRI texture and, using the formulas (1, 2), accumulates the final color. C is the color, A is the opacity. Since the rectangle representing the screen is located at an arbitrary point in space, for a faster calculation it is recommended to additionally calculate the first point of intersection of the ray with the unit cube and define it as the starting point of the ray. When changing the position of a cube or screen rectangle, it is necessary to recalculate the first intersection of the ray with the unit cube. (1) (2)

B. Tissue surface rendering method.
This method is similar to direct volume rendering. The only difference is that the ray stops immediately upon collision with the target tissue, and the tissue opacity is 1, i.e. maximum. This makes it easy to render the surface. This makes it easy to display the surface. This method also needs to compute the shading that requires surface normal. The surface normal of a 3D texture can be computed as minus the gradient of the scalar data. An example of how this method works is shown in the figure 1.

V. BRAIN EXTRACTION
To display the data on the brain, it must be extracted from the MRI data. The task of isolating the brain is difficult, as it is based on the input MRI data, which varies over a wide range. We distinguish the following approaches to this task: It is the most common way to isolate various tissues, including the brain. Often, the developers of MRI devices provide a set of ranges of values corresponding to a particular tissue, which allows you to immediately fill a one-dimensional transfer function according to already known conditions. If the developers do not provide ranges of values or they are not enough, you must select them ~ 12 ~ manually. For this, a one-dimensional histogram is calculated, where each bin stores the value of the number of voxels with a certain scalar value, and then peaks corresponding to scalar data are selected on this histogram. At the same time, to increase the severity of peaks, it is recommended to use a logarithmic scale, rather than a linear one.
B. Two-dimensional transfer function.
The use of a two-dimensional transfer function is an extension of the one-dimensional transfer function approach. To construct the transfer function in this method, a two-dimensional histogram is used, where the values of the derivative along the gradient are located along the second axis, which is simply the magnitude of the gradient. The advantage of this method is that it allows you to select those tissues that are indistinguishable when using a onedimensional histogram [7]. It should be remembered that this method is not an ultimatum. It does improve quality, but it may not be enough.
The 2D transfer function allows you to highlight the boundaries, which are expressed by arches on the 2D histogram.

C. Ready-made automatic algorithms.
The set of these methods consists of off-the-shelf brain extraction tools such as ROBEX, BET, AFNI and others. They are well described in article [8]. This approach has the advantage of being quick and easy to work with, and it often allows you to isolate data that cannot be extracted using transfer function methods. However, despite the fact that these methods are called automatic, sometimes they still require manual processing. For example, BSE has problems with the separation of the brain and neck. However, the greatest disadvantage of these methods is that they are black boxes and may return incorrect results for brains with any physical disabilities or not suitable for working with the brain of an animal. Although the same problems are observed with transfer function methods, they are less pronounced.

A. Depth loss effect
The term "depth" refers to the distance that the ray travels within a selected area, for example, inside a tissue. The smaller the width of the tissue in the direction of the ray, the less depth this tissue has in relation to the ray. This is manifested by the loss of perception of the geometry of the object. An example of this is shown in figure 3 (а). This is a consequence of the fact that the method of two-dimensional transfer function is characterized by the selection of boundaries between tissues, and not the tissues themselves, which have a single or small depth and at the same time do not describe the volume of the tissue in the direction of the ray. Greater depth has the advantage that the ray, passing through it, accumulates its constant optical properties and changes the color it stores accordingly. This allows you to visually perceive relative color changes and interpret through this distance to another tissue ( fig. 3 (c)) shows an example of using the one-dimensional transfer function method, which retains the depth effect, since the transfer function highlights the entire tissue, including its interior. Figure 3 (d) shows an enlarged segment of this example. As you can see, with the expansion, the pixel color is shifted to red. Thanks to this, the human brain can perceive volume.
To solve the problem of the effect of depth, it is necessary to perceive the boundaries as smooth surfaces and use the shading method. Figure 3 (b) shows an example of the impact of this approach. In this example, we used Phong shading. As you can see, this improves the perception of the geometry. The blue arrow shows the direction of muscle depth increasing. Rays become redder in areas with a high depth of muscle.

B. Calculation of the normal.
To apply shading to an image, you need to know the surface normal. The surface normal in volume data can be calculated as minus the normalized gradient. Therefore, the task is to calculate the gradient. The most popular method for calculating gradient in volume data is the centerweighted gradient method. It is calculated as the sum of the products of the scalar values of neighbouring voxels by the vector direction to these voxels. You can take into account only 6 neighbours, which are located on the axes of 3D texture coordinates, or 26 neighbours, adding those located on the diagonal. Figure 6 (a) shows an example of shading ~ 13 ~ using a 6-point center weighted gradient, Figure 6 (b) shows an example of 26 points. When calculating the gradient, there is a problem of anisotropy, since often in MRI data voxels along one of the axes have a different physical length from the rest. This leads to the fact that a discrete step along this axis leads to a larger or smaller step in physical space with respect to other axes. There are two approaches to this problem: 1. Linear data interpolation. 2. Allow the same spatial resolution. Figure 4 shows an example of how these two approaches work. The difference between them is insignificant and is expressed only in the intensity of shading, so any approach is allowed.

C. Problem of the center-weighted gradient method
There is a center-weighted gradient calculation method problem. Imagine a certain region of heights in continuous space, where the entire surface is flat, excluding a single peak, and the area of the upper part of this peak tends to zero ( Figure 5 (a)). Let us describe this region of heights with a two-dimensional discrete texture, and such that the physical length and width of each pixel is less than the length and width of the change in the height of the peak. In this case, the pixel describing the peak will store the average height. An example of such a height map is shown in Figure 5 (b). When calculating the gradient magnitude of such a height map by the center-weighted method, the gradient texture will be obtained, shown in the figure 5 (c). On it, high values of the gradient magnitude enclose the peak in a ring, however, referring to the original data in continuous space, the gradient is equal to zero around the peak. The problem is that the gradient exists between discrete elements, but not within them. This is a consequence of the physical limitation of the mapping of continuous space to discrete one.

D. Node gradient calculation method
To solve this problem, we propose to calculate the gradient at the nodes between pixels (or voxels for 3D data). This will allow you to more accurately calculate the gradient. A natural problem with this approach is that no scalar value is defined for the nodes. We propose to calculate it as an average between data items. surrounding the node. Figure 6 shows an example of rendering a head surface using three different methods. As you can see, the 26-point center-weighted method gives the smoothest result. Node gradient calculation method is good at highlighting slices of the MRI texture.
Based on the nature of these methods, we can distinguish the following approaches to calculating the gradient, highlighted by the importance of the data: 1. scalar > gradient important approach 2. gradient > scalar important approach The choice of a specific method is determined by the formulation of the problem. This paper uses a 26-point center-weighted gradient method.

E. Convolution in removing noises
The presence of noise in the MRI image is one of the rendering problems. One way to solve this problem is to use convolution. To use this method, you need to create a convolution kernel. For the three-dimensional case, the kernel is specified as a symmetrical three-dimensional cubic ~ 14 ~ texture. For filling the kernel, the most popular distribution is Gaussian distribution (3).

(3)
A kernel is applied to each element of the texture. The central value in the kernel is multiplied by the value of the element to which the convolution is applied. The rest of the elements in the core are multiplied by their neighbours in accordance with the spatial arrangement of the core and texture elements. The return value is the sum of all these products. It is important to remember that for the current task the sum of the elements of the convolution kernel must be equal to one, i.e. the kernel must be normalized. In this case, it is not necessary to use a large core, as practice shows, a core from 3x3 to 7x7 is sufficient. The sigma in the Gaussian equation determines the degree of smoothing. If the kernel is based on a Gaussian distribution, then such a convolution results in smoothing of data outliers, which are noise. Figure 7 shows examples of such a convolution, where (a) textures without convolution, (b) textures with convolution. For clarity, a different set of threshold values was used, where the threshold value determined the minimum scalar value in the MRI texture that was not ignored by the ray. As you can see, at all three threshold values of the MRI, the texture after convolution shows less noise. The disadvantage of this approach is that it smoothes the sharpened surface, which leads to a decrease in the visibility of convolutions, etc.
The kernel does not have to be limited to a Gaussian distribution. You can try using different distributions, filters or breaking symmetry. This provides opportunities for further research.

VII. EEG MAPPING
EEG is a non-invasive method for studying the electrical activity of the surface of the brain. To display EEG data, it is necessary to set the position of these sensors relative to the brain.

A. Installation of EEG sensors.
It is proposed to set them manually on the MRI image render. It is convenient to do this by some implementation of the casting algorithm. Since the full MRI texture is available, it is convenient to position the sensors not relative to the brain, but relative to the surface of the head. This gives a better idea of their location.: The user selects one of the sensors and clicks on the head's render.
Knowing the initial position of the ray, calculated from the cursor position, and the direction of the ray, the collision point can be calculated.
To calculate the collision point, you can use Bresenham's algorithm for the three-dimensional case, or iterate with a constant step in the MRI texture space. Figure 8 shows an example of installed sensors on the head surface.

B. Display of instant EEG data
To present EEG data, it is proposed to use an approach in which an element of the brain surface corresponds to electrical activity, which is the closest sensor, but with distances no more than sigma, where sigma is a constant that depends on the EEG device. An example of such a display is shown in Figure 9. The color determines the sign, and the magnitude of the value recorded by the sensor is displayed as color intensity. This approach allows you to see the formation of domains on the surface of the brain and assess the propagation of waves.
An approach was also considered in which the intensity depended also on the distance to the sensor, decaying at a distance from it. However, this approach gives a poorly understood image and is characterized by signal mixing.
An approach was also considered in which the signal intensity was linearly interpolated between the nearest sensors, but the problem with this approach is that for correct interpolation, the distance between the sensors on the surface of the brain, and not in volume, is necessary. Fig. 9. Displaying of instant EEG data. Green is a positive value, blue is a negative value.

VI. RESULTS AND CONCLUSIONS
In this paper, a method for displaying EEG data on the surface of the brain was considered. The brain was extracted with an automatic algorithm that provided a voxel mask of the corresponding tissues. Depending on the quality of the data, one-dimensional or two-dimensional transfer function methods can be used for this task. The render shows that domains are formed on the surface of the brain -areas of electroactivity with a similar sign and value.This study can be improved by adding a display of the total energy that was recorded by each sensor over the last few ticks. This will improve the perception of active areas of the brain. An overview of the effect of Gaussian convolution, which reduces noise, was also presented. There is a way for further work in this direction. You can study the effect of different convolution kernels on noise smoothing, brain selection, and more.
Node gradient calculation method was presented. This is an interpretation of the center-weighted gradient calculation method. This method can be applied in various areas of modeling. It has the advantage of a more accurate gradient calculation, but it has the disadvantage of losing the originality of the original data.
Increasing the density of the sensors can show a more accurate picture. Of particular interest is the improvement of resolution near the arteries. This improvement could show the effect of blood flow on the electrical activity of the cerebral cortex.