Posts | Hugo Ninou

Paper deep dive : Linking connectivity, dynamics and computations in low-rank recurrent neural networks, Mastrogiuseppe & Ostojic, 2018

Sun, 08 Sep 2024 15:36:22 +0000

Introduction
Theoretical Framework
1. Building a firing rate model
  1. The firing rate
  2. The total synaptic current
2. Networks with low-rank connectivity matrices
Spontaneous Activity
1. Reproduction of the paper’s phase diagram
2. Comparison with the phase diagram in the $\tau_r \gg \tau_s$ case
Response to an external input
1. Reproduction of figure 2.D of the article
Conclusion

Introduction

Cortical networks, which consist in highly interconnected neurons with recurrent synapses are believed to make for the fundamental units of mammalian brains. Observations show that cortical connectivity lies somewhere between fully structured and fully random. Several functional approaches have been made for connectivity design of cortical networks since the 80’s but they lack a unifying conceptual picture. To address this matter, authors point out that all these approaches share something in common: the fact that the resulting connectivity matrices are low rank. This article [1] aims at linking the recurrent neural networks’ dynamics to their connectivity matrix and showing how one can design the low-rank connectivity structure of such networks to implement specific computations. The latter point is illustrated on four specific tasks.

Theoretical Framework

The model used here to describe cortical neural networks is a firing rate model. This means that each node (i.e. neuron) in the network is represented by its firing rate $\phi(x_i)$ with $\phi(x)=\textrm{tanh}(x)$ being the current-to-rate transfer function, and $i\in [1\dots N]$, with $N$ the number of neurons. The evolution of a neuron’s firing rate is governed by equation (1).

$$ \dot{x_i}(t) = -x_i(t) + \sum_{j=1}^N J_{ij} \phi(x_j(t)) + I_i \quad (1) $$

where $J_{ij}$ is the connectivity matrix representing the synaptic connections of the network and $x_i$ is the external current input to neuron $i$. Note here that $\phi(x_i)$ representing the firing rate can have negative values. This can be dealt with by replacing the $\textrm{tanh}$ function, which makes the calculations easier, by a sigmoid without causing major changes to the theoretical results.

As all the results of this article come from analysis and simulations of this model, it is crucial to understand its limitations as for the description of cortical neural networks. With this goal in mind, we explore some developments in order to understand the underlying hypotheses and limitations of this model by building it again from scratch [2].

Building a firing rate model

The behaviour of a neuron can be described by the neuronal response function $y(t)$ that encodes the exact time at which it fires spikes. A model involving $y(t)$ is called a spiking model. $$ y(t) = \sum_{i=1}^n \delta(t-t_i), \quad r(t) = \int_t^{t+\Delta t} \underbrace{\frac{1}{\Delta t}\langle y(\tau) \rangle d\tau}_{\textrm{average over the trials}} \quad (2) $$

Firing rate models focus on the quantity $r(t)$ in eq. (2) which is an approximation of the exact spike sequence $y(t)$. They have the advantage of being easier to simulate on computers as they do not take into account the short time scale dynamics of the spikes. As we want to model the total input for the neurons, we can look at $r(t)$ instead of $y(t)$ if there is not too much variability between two trials. Indeed, upon summing over different synapses, one has low variability (Central Limit Theorem) if the entries are numerous and uncorrelated.

Firing rate models are relevant when

The firing of neurons in a network is uncorrelated (there is little synchronous firing)
The precise patterns of spike timing are unimportant. Indeed, the information regarding those precise patterns is lost when averaging over the trials.

Figure 1: Sketch of a modeled neuron with presynaptic inputs $r_i$ and postsynaptic output $r(t)$. $x(t)$ is the total synaptic current or input.

In order to fully describe a firing rate model, we have to specify the dependence of the postsynaptic firing rate on the total synaptic input $r(x)$ and the dependence of the total synaptic input on the presynaptic inputs $x(r_1,r_2,r_3,r_4)$.

The firing rate

Let’s first describe the firing rate $r(t)$ as a function of the total synaptic current $x(t)$. We could simply write $r(t) = \phi(x(t))$, $\phi$ being the current-to-rate function, but due to the membrane capacitance and resistance, we should rather express the firing rate $r(t)$ as a low-pass filtered version of its steady state with characteristic time $\tau_r$, usually of the order of 20 ms.

$$ \tau_r \frac{dr}{dt} = -r + \phi(x(t)) $$

Note that in reality, it is the membrane potential, not the firing rate, that is a low pass of the input current and that the dynamics of the two are not the same.

The total synaptic current

We now want to write the total synaptic current of neuron $i$, $x_i(t)$ as a function of the $N$ presynaptic firing rates $r_j(t)$ for $j \in [1 \dots N]$ and their associated weights $J_{ij}$. Note that $J_{ij}>0$ corresponds to an excitatory synapse while $J_{ij}<0$ corresponds to an inhibitory one. We introduce the synaptic kernel response function $K(t)$ that is simply the response current induced at time $t$ by a spike at time $t=0$. Assuming that the effects of a spike sum linearly, we can then write the total synaptic current as

$$ x_i(t) = \sum_{j=1}^{N} J_{ij} \int_{-\infty}^{t} d\tau K(t-\tau) \underbrace{y_j(\tau)}_{\sum_{k=1}^{N} \delta(\tau-t_k)} $$

which in the firing rate model approximation writes

$$ x_i(t) = \sum_{j=1}^{N} J_{ij} \int_{-\infty}^{t} d\tau K(t-\tau) r_j(\tau) \quad (3) $$

By taking $K(t) = \exp(-t/\tau_s)/\tau_s$ (with $\tau_s$ the time constant that describes the decay of the synaptic conductance, usually of the order of a few milliseconds), then (3) can be written as a differential equation

$$ \tau_s \frac{dx_i(t)}{dt} = -x_i(t) + \Big( \sum_{j=1}^{N} J_{ij} r_j(t) + I_i\Big) \quad (4) $$

This is actually equation (1) of the paper in which $\tau_s$ was chosen equal to 1 for simplicity. Equations (4) and (2) give us the two parts needed to describe a firing rate model which can be simplified in two extreme cases:

$\tau_r \gg \tau_s$: Then $x_i(t) = \sum_{j=1}^{N} J_{ij} r_j(t) + I_i$ and $r$ is a low-pass of $x(t)$.
$\tau_r \ll \tau_s$: Then $r(t) = \phi(x(t))$, $r(t)$ follows $x(t)$ instantaneously.

Authors implicitly consider that we’re in the situation where $\tau_r \ll \tau_s$ which is not obvious at all as both characteristic times seem to be of the same order of magnitude. This observation led me to explore what would have been the paper’s results if they had instead considered the situation where $\tau_r \gg \tau_s$. This latter case yields the following equation for the system that has to be compared to equation (1):

$$ \dot{r_i} = -r_i + \phi(\sum_{j=1}^N J_{ij}r_j+I_i) \quad (5) $$

In the following, we will reproduce some of both theoretical and simulatory results of the paper and try to extend them to a system governed by equation (5).

Networks with low-rank connectivity matrices

We start by placing ourselves in the same context as in the article i.e. with a negligible membrane relaxation characteristic time.

The connectivity matrix $J_{ij}$ is the sum of an uncontrolled random matrix $\chi$ and of a structured low ranked known matrix $P$. $J_{ij}$ is thus defined by

$$ J_{ij} = \underbrace{g \chi_{ij}}_{\textrm{mean $0$, variance $g^2/N$}} + \underbrace{P_{ij}}_{\textrm{of order $1/N$}} $$

Note that there is no biological reason for which $\chi_{ij}$ should have a variance scaling as $1/N$. This constraint makes possible the comparison of networks of different sizes from a theoretical perspective (especially in the case $N \to \infty$) and can be dealt with by adjusting the random strength $g$ at will.

Networks with unit-rank structure

We start with $P_{ij} = \frac{m_i n_j}{N}$ with $m=\{m_i\}$ and $n=\{n_j\}$ two N-dimensional vectors. Authors define two important parameters, $g$ the random strength, and $m^Tn/N$ the structure strength, that govern the type of dynamics of the system. The type of networks studied here is related to the Hopfield networks studied in class. However the unit-rank terms here do not require to be symmetric and can be correlated to each other. Regarding the biological plausibility of this proposed connectivity, it can be noticed that Dale’s law is not imposed here.

Dynamical Mean-Field Theory

Under the assumption of a large network with a weak low-dimensional connectivity matrix (scaling as $1/N$) one can derive the activity of each neuron thanks to dynamical mean-field theory by considering the mean and variance of the input it receives. Authors find that the average equilibrium input to unit $i$ is denoted $\mu_i = \kappa m_i$ with $\kappa = \langle n_i[\phi_i] \rangle_i$, that is that the activity of the network is one dimensional, along the vector $m$ as long as $\kappa>0$. As $\kappa$ represents the activity projected on vector $n$, non-vanishing values of $\kappa$ require a non-vanishing overlap between $m$ and $n$.

Dynamical Mean-Field Theory extension to the $\tau_r \gg \tau_s$ case

Starting from equation (5), we derive a Dynamical Mean-Field approach in order to express both $\mu_i \equiv [x_i]$ and $\Delta_0^I \equiv [x_i^2] - [x_i]^2$. Similarly to the derivation proposed in the supplementary information of the paper, one can consider the case where $I_i=0$ \forall i. By denoting

$$ \eta_i(t) = \sum_{j=1}^N J_{ij} r_j = g \sum_{j=1}^N \chi_{ij} r_j + \frac{m_i}{N} \sum_{j=1}^N n_j r_j $$

equation (5) can be rewritten as

$$ \dot{r_i} = -r_i + \phi(\eta_i) $$

In the stationary scenario, we would then have

$$ r_i = \phi(\eta_i) $$

By applying $\phi^{-1}$ to both sides of this equation, we fall back on equation 28 of the paper that gives us an expression for $\mu_i$ and $\Delta_0^I$.

$$ \mu_i = [x_i] = m_i \kappa $$ $$ \Delta^I_0 = [x_i^2] - [x_i]^2 = g^2 \langle [\phi_i^2] \rangle $$

Conducting a DMF analysis in the chaotic scenario is however trickier, and we do not develop it in this project although it is an interesting lead.

Spontaneous Activity

Reproduction of the paper’s phase diagram

Figure 2: Phase diagram obtained from personal simulations (50×50 resolution). Left: measures the chaotic nature of the system. Right: measures the structured nature of the system.

Figure 3: Left: Theoretical phase diagram from the article. Right: Phase diagram obtained from personal simulations (50×50 resolution).

In figure 1 of the article, a phase diagram is proposed. The result presented shows the phase diagram obtained from theoretical results. We chose to reproduce this phase diagram thanks to simulations (fig.3). To do so, finding the good statistics to describe the chaoticity and structuration of the system is crucial. Authors hint us in using the temporal variance of $x_i$, averaged over the number of neurons, to characterize chaoticity $\langle \textrm{std}_t(x_i) \rangle_i$. We use the activity along $m$, given by $\kappa = \langle n_i[\phi_i] \rangle_i$ to characterize structure. We superimposed the phase diagrams for chaoticity and structure shown in figure 2 to obtain the one shown on the right panel of figure 3.

Each of the 2500 pixels on the simulated phase diagram represents a simulation. The statistics were measured after a transient phase was over for 100 seconds with parameter $\Delta t = 0.5 \textrm{s}$. Considering the ergodicity theorem, this allows us not to have to make several simulations for each pair of parameters, which would have multiplied by the same amount the time spent in simulating the transient phase.

The biologically plausible phase is the one where there is both structured and chaotic activity.

Comparison with the phase diagram in the $\tau_r \gg \tau_s$ case

We investigate the changes that using equation (5) instead of (1) would bring to the phase diagram describing the system’s behavior. Interestingly, the phase diagram obtained for equation (5) is very similar to that of (1) (see figure 4). This was expected for the stationary part of the diagram but not necessarily for the chaotic part. This result hints us into thinking that the DMF theoretical results one would derive for our alternative model might be the same as the ones found in the paper.

Figure 4: Left: Phase diagram for $\tau_r \ll \tau_s$. Right: Phase diagram for $\tau_r \gg \tau_s$ (resolution 50 × 50).

Response to an external input

For further analysis, we aim at comparing the behavior of our alternative model to that of the paper in response to an external stimulus $I$. As in the article, we look at the response of the system along the $m$ vector. Figure 5 shows the transient dynamics for both models when the system is subject to an external input with the same connectivity matrix and the same initial conditions. One can see that, as expected by the theory, the stationary states are the same while the transient phases slightly differ. Indeed, the system seems to evolve more slowly for the alternative model (figure 5 right).

Figure 5: Transient dynamics in the $\tau_r \ll \tau_s$ (Left) and $\tau_r \gg \tau_s$ (Right) scenarios in response to a step input.

Reproduction of figure 2.D of the article

Figure 6, corresponding to figure 2.D of the article, plays a major role in the four task implementations designed in the following of the paper. Reproducing this result for our alternative model basically ensures that it will be able to perform the proposed tasks as well. The reproduction of this result is presented in figure 7. The activity along vector $m$ is recorded for 15 different intensities of the input in three case scenarios. First, in the case where $m$, $n$, and $I$ are all orthogonal in respect to each other. The system hence shows no response along $m$ as $\kappa = \langle n_i[\phi_i] \rangle_i = 0$ (fig.7 left). Second, in the case where $m$ and $n$ are orthogonal and $I$ has a component along $n$. The network then shows some activity along $m$ (fig.7 center). Finally, in the case where $m$ and $n$ have an overlap and $I$ is colinear to the orthogonal part of $n$ with respect to $m$, the system shows a bistable activity along $m$ (fig.7 right). This bistable activity vanishes when the input becomes too strong.

The same results were observed for our alternative model, and are presented in figure 8.

Figure 6

Figure 7: Activity along $m$ as a function of input strength in three different scenarios in the $\tau_r \ll \tau_s$ scenario. Left: $m$, $n$, and $I$ are orthogonal. Center: $m$ and $n$ are orthogonal, and $I$ has a component along $n$. Right: $m$ and $n$ have an overlap.

Figure 8: Same as figure 7 but in the $\tau_r \gg \tau_s$ scenario.

Conclusion

Thanks to a carefully conducted critical reading, we noticed that when building their firing rate model, authors implicitly did not explore an alternative model in which $\tau_r$ (the membrane characteristic time) is not negligible with respect to $\tau_s$ (the synaptic characteristic time). This alternative model is an interesting extension as it covers a theoretical framework ignored by the authors. We showed through a short theoretical analysis that both models were equivalent in the stationary case. Finally, we showed experimentally that the alternative model we proposed in equation (5) has very similar behavior to that of the paper’s, both for spontaneous activity and in response to an external stimulus. Because this behavior analysis makes for the building block of the four task implementations, it is legitimate to think that the proposed alternative model will also be able to perform these tasks. The mini-project conducted here therefore shows that the low-rank recurrent neural networks studied in this paper are even more biologically plausible than they claim to be.

References

[1] Francesca Mastrogiuseppe and Srdjan Ostojic. Linking connectivity, dy-namics and computations in low-rank recurrent neural networks. Neuron, 99(3):609–623.e29, August 2018. arXiv: 1711.09672.
[2] Peter Dayan and L. F. Abbott. Theoretical neuroscience: computational and mathematical modeling of neural systems. Computational neuro-science. Massachusetts Institute of Technology Press, Cambridge, Mass, 2001.

Data challenge : Can you predict the tide ?

Mon, 01 May 2023 07:04:36 +0000

Introduction
Data Analysis
1. Heat Maps
2. Autocorrelation of the Tidal Surplus Over Time
Development of a Recurrent Neural Network (RNN)
Conclusion

Introduction

This project is part of the course Information and Complexity and aims to address the challenge ‘Can you predict the tide?’. The challenge, proposed by the FLUMINANCE team at Inria, consists of predicting the tidal surplus based on past measurements and pressure fields. A direct application of this challenge is to provide more efficient responses to extreme tidal events. In this report, we detail our approach, which is carried out in two stages: we first present key elements of the data analysis, and then propose a relevant implementation of an Encoder-Decoder model to meet this challenge.

Data Analysis

Heat Maps

To propose the most relevant algorithms to solve the given problem, we started by studying the dataset. A first approach consists of performing a linear regression of the tidal surplus at a given time based on each point of the pressure field at the same time. Prior to this, the values of the pressure field were z-scored. The correlation between the tidal surplus and the pressure field at a point provides us with the heatmap in Figure 1. Note that there are 5 times more pressure fields than time points of tidal surplus. Therefore, linear regression is done by taking the pressure field closest in time to the tidal surplus. Blue areas correspond to points where a depression induces a positive tidal surplus, and conversely, red areas show the opposite. From this simple analysis, we can already suggest that City 1 is likely located in the Southeast quadrant of the map, while City 2 is likely in the Northwest quadrant.

We can proceed similarly using the spatial derivatives of the pressure field, which we refer to as horizontal wind (derivative along the x-axis) and vertical wind (derivative along the y-axis). The results are shown in Figures 2 and 3. Overall, the horizontal wind appears to carry more information about the tidal surplus (the heat maps for the vertical wind contain many coefficients close to zero).

Figure 1: Heatmap of the correlation between the centered and reduced pressure field and the tidal surplus for City 1 (left) and City 2 (right).

Figure 2: Heatmap of the correlation between the centered and reduced horizontal wind and the tidal surplus for City 1 (left) and City 2 (right).

Figure 3: Heatmap of the correlation between the centered and reduced vertical wind and the tidal surplus for City 1 (left) and City 2 (right).

Autocorrelation of the Tidal Surplus Over Time

We have briefly discussed the information that the pressure fields provide regarding the tidal surplus in Cities 1 and 2. In this section, we analyze the information that the time series of the tidal surplus carries about itself. One way to do this is by looking at the autocorrelation of the tidal surplus over time for Cities 1 and 2. After reordering the tidal surplus data in chronological order, we plotted the autocorrelation curves shown in Figure 4. One limitation of this approach is that the time intervals between consecutive points are irregular. On average, the interval is 9 hours, but about two-thirds of the points are evaluated at the same time as the previous point, likely due to an expansion of the dataset. Nonetheless, it is still possible to observe that the tidal surplus is quite autocorrelated over time, with a more marked correlation for City 2 than for City 1. Therefore, it will likely be much easier to infer accurate results for City 2 than for City 1, which is indeed what we observe when separating the prediction scores for each city.

Figure 4: Autocorrelation of the tidal surplus for City 1 (left) and City 2 (right).

Development of a Recurrent Neural Network (RNN)

The development of artificial neural networks for solving prediction problems in meteorology and climatology is relatively recent. Here, we draw inspiration from two articles that use an Encoder-Decoder architecture [1][3], which have demonstrated state-of-the-art performance for time series prediction.

The Encoder-Decoder Model

The architecture of the Encoder-Decoder model that we implemented is shown in Figure 5. It consists of a sequence of recurrent cells (GRU cells) that take as input a vector from a time series and pass a “hidden vector” encoding the system state. Once the entire input sequence has been given to the encoder, the final “hidden vector” is passed to a decoder made of recurrent cells (decoder cells), which take the tidal surplus at time (t) as input and predict the tidal surplus at time (t+1). The implementation is done in Python using the Pytorch library.

Regarding recurrent cells, several choices can be made. The basic cell consists of a simple matrix multiplication followed by the application of a non-linear function. This approach presents issues with learning (gradient explosion or vanishing gradients [4]). To address this, LSTM cells were developed. GRU cells are an alternative, offering fewer parameters while providing identical performance.

The gradient descent is performed using the Adam [5] method with L2 regularization.

Figure 5: Architecture of the implemented Encoder-Decoder model.

Splitting the Data into Training Set and Testing Set

We started by dividing the dataset into a training set (90%) and a testing set (10%) randomly. However, large discrepancies were observed between the testing set score we created and that of the challenge. This was due to the fact that the dataset contains several points that are actually very close or sometimes overlapping in time, causing overfitting on the training set to affect the testing set. To remedy this, we split the dataset into blocks of consecutive time points that are thus correlated over time (refer to Figure 4).

Naive Implementation Without Using Pressure Fields

A first attempt was made by using the neural network as presented in Figure 5, but only taking the time series of tidal surplus as input and ignoring the pressure and wind fields. To minimize the score, we fine-tuned the network’s parameters, such as the number of layers within each cell, the size of the hidden vector, and the dropout rate (randomly ignoring a certain percentage of parameters during training). We found that with two layers, a hidden vector of size 25, and a dropout rate of 20%, we achieved a minimum score of 0.63. The evolution of the test score during training is shown in Figure 6.

Figure 6: Scores for the training set and test set over the epochs of learning in the naive implementation without using pressure fields.

Naive Implementation Using All Pressure and Wind Fields

A second approach involved constructing a large vector comprising the tidal surplus in each city, along with the pressure and wind fields at the closest time (see Figure 7). The results obtained in this case are shown in Figure 8, using four layers, a hidden vector of size 200, and a dropout rate of 50%. This network achieved a minimum score of 0.57, which is a 5% improvement over the previous method but at the cost of a 25,000-fold increase in the number of parameters. A significant limitation of this implementation is the tendency for the model to overfit, despite L2 regularization, which forced us to use a very high dropout rate.

Figure 7: Architecture with expanded input vectors including pressure and wind fields.

Figure 8: Scores for the training set and test set over the epochs of learning when using all pressure and wind fields.

Implementation with Dimensionality Reduction of Pressure and Wind Fields

Finally, we propose a less naive implementation of the neural network that leverages our data analysis. Instead of feeding the entire pressure and wind fields into the model, we compute a dot product between these fields and their associated heat maps. A field of size (41 x 41=1681) is reduced to a vector of size 2 (one dimension per city). The input vector’s size is thus reduced from 5045 to 8. With four layers for the recurrent cells and a hidden vector of size 25, we achieved a score of 0.5 (Figure 9, right), representing an 8% improvement over the naive method that included raw pressure and wind fields. This proposed method is more efficient and resource-conservative, requiring 5000 times fewer parameters.

Figure 9: Scores for the training set and test set over the epochs of learning with dimensionality reduction of pressure and wind fields.

Note that we are only using one-fifth of the available pressure fields. An extension to consider would be to not only use the pressure field closest in time to the tidal surplus measurement but also include the 4 previous fields, which could provide additional information. In this case, it would be necessary to compute heatmaps showing the correlation between the tidal surplus at time (t) and these fields at times (t-1), (t-2), (t-3), and (t-4).

Conclusion

Based on a thorough data analysis, we proposed an Encoder-Decoder model to tackle this challenge. In the context of the Encoder-Decoder model, the issue of compressing the information contained in the pressure field is central. Feeding the network with raw pressure fields leads to overfitting problems, which are difficult to resolve. The method we propose, which reduces entire fields to a single scalar, may be extreme but offers good explainability and yields very good results. The optimal approach likely lies somewhere between these two extremes, avoiding translation-invariant compression methods, as the location of depressions and winds is crucial for solving the problem.

References

[1] Sebastian Scher and Gabriele Messori. Weather and climate forecasting with neural networks: using general circulation models (GCMs) with dif- ferent complexity as a study ground. Geoscientific Model Development, 12(7):2797–2809, July 2019.
[2] Nathawut Phandoidaen and Stefan Richter. Forecasting time series with encoder-decoder neural networks. arXiv:2009.08848 [math, stat], September 2020. arXiv: 2009.08848.
[3] Encoder-Decoder Model for Multistep Time Series Forecasting Using Py- Torch | by Gautham Kumaran | Towards Data Science.
[4] Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training Recurrent Neural Networks. arXiv:1211.5063 [cs], February 2013. arXiv: 1211.5063.
[5] Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Opti- mization. arXiv:1412.6980 [cs], January 2017. arXiv: 1412.6980.

Posts | Hugo Ninou

Paper deep dive : Linking connectivity, dynamics and computations in low-rank recurrent neural networks, Mastrogiuseppe & Ostojic, 2018

Table of Contents

Introduction

Theoretical Framework

Building a firing rate model

Figure 1: Sketch of a modeled neuron with presynaptic inputs $r_i$ and postsynaptic output $r(t)$. $x(t)$ is the total synaptic current or input.

The firing rate

The total synaptic current

Networks with low-rank connectivity matrices

Networks with unit-rank structure

Dynamical Mean-Field Theory

Dynamical Mean-Field Theory extension to the $\tau_r \gg \tau_s$ case

Spontaneous Activity

Reproduction of the paper’s phase diagram

Figure 2: Phase diagram obtained from personal simulations (50×50 resolution). Left: measures the chaotic nature of the system. Right: measures the structured nature of the system.

Figure 3: Left: Theoretical phase diagram from the article. Right: Phase diagram obtained from personal simulations (50×50 resolution).

Comparison with the phase diagram in the $\tau_r \gg \tau_s$ case

Figure 4: Left: Phase diagram for $\tau_r \ll \tau_s$. Right: Phase diagram for $\tau_r \gg \tau_s$ (resolution 50 × 50).

Response to an external input

Figure 5: Transient dynamics in the $\tau_r \ll \tau_s$ (Left) and $\tau_r \gg \tau_s$ (Right) scenarios in response to a step input.

Reproduction of figure 2.D of the article

Figure 6

Figure 7: Activity along $m$ as a function of input strength in three different scenarios in the $\tau_r \ll \tau_s$ scenario. Left: $m$, $n$, and $I$ are orthogonal. Center: $m$ and $n$ are orthogonal, and $I$ has a component along $n$. Right: $m$ and $n$ have an overlap.

Figure 8: Same as figure 7 but in the $\tau_r \gg \tau_s$ scenario.

Conclusion

References

Data challenge : Can you predict the tide ?

Table of Contents

Introduction

Data Analysis

Heat Maps

Figure 1: Heatmap of the correlation between the centered and reduced pressure field and the tidal surplus for City 1 (left) and City 2 (right).

Figure 2: Heatmap of the correlation between the centered and reduced horizontal wind and the tidal surplus for City 1 (left) and City 2 (right).

Figure 3: Heatmap of the correlation between the centered and reduced vertical wind and the tidal surplus for City 1 (left) and City 2 (right).

Autocorrelation of the Tidal Surplus Over Time

Figure 4: Autocorrelation of the tidal surplus for City 1 (left) and City 2 (right).

Development of a Recurrent Neural Network (RNN)

The Encoder-Decoder Model

Figure 5: Architecture of the implemented Encoder-Decoder model.

Splitting the Data into Training Set and Testing Set

Naive Implementation Without Using Pressure Fields

Figure 6: Scores for the training set and test set over the epochs of learning in the naive implementation without using pressure fields.

Naive Implementation Using All Pressure and Wind Fields

Figure 7: Architecture with expanded input vectors including pressure and wind fields.

Figure 8: Scores for the training set and test set over the epochs of learning when using all pressure and wind fields.

Implementation with Dimensionality Reduction of Pressure and Wind Fields

Figure 9: Scores for the training set and test set over the epochs of learning with dimensionality reduction of pressure and wind fields.

Conclusion

References