You and I – 5

I: I think I am full of superiority complex. I have this condescension and self-righteousness as I can see through. And why won’t I? I feel disgusted when I can see the stupid self-destructive actions people take. Why are people so stupid?

You: It’s an illusion. You come from what you have experienced. You may know certain things more. But you don’t know, can not know everything as you have not experienced everything. You only know your life and you see everything through your lens. You may be right in terms of what is to be done right, but you have no clue why they might not have been understanding what you understand.

I: I hate to fail.

You: It’s ok if you fail now. Accept the failure. It does not mean “You are THE FAILURE”.

I: I hate deadlines.

You: Deadlines are there so that you can enjoy the freedom afterward. Ultimately you are free and there is no rush. But don’t you want to accomplish things in life? If you do, then deadlines are your friends.

I: Is my physical experience that important?

You: Your physical appearance, experience are important. Because your mind lives inside your physical body.

I: Why am I so much bothered by the physical cravings – hunger for food, sex etc? They are killing me.

You: They are your physical needs which need to be dealt with for your mind to keep engaged and flourish. Neglecting them, you will starve yourself and feeling starved your mind will not let you who you can truly become.

I: Aren’t physical appearances just the way to attract other people?

You: Yes, but it depends on how you look. Our reality is that we already live in a society. You won’t be who you are if your parents wouldn’t raise you properly, if your closed ones wouldn’t love and take care of you. However, attracting other people may not be the final goal of life. Therefore, if you are in a place where your physical appearance is harming your mental peace- like if you are fat and feel shame, you are creating a problem for your own self. There are things you can try to reconcile with. You need to think about your priorities – what you really care about. But you need to believe that you can change. You should be able to hope. You are more than your body and your mind. So say when you feel vulnerable “I am not just my body. I am not just my mind either.”

I: What do you think about minimalistic life- making everything simple, possessing few things?

You: It might help you stay away from debt or pay off your debt. Letting redundant things go can save yourself from taking unnecessary burdens Depending on how you do it, it can be adventurous. You can be a digital nomad or an urban nomad. You can travel around, make more friends and have some exciting experiences. But you need to be brave and be ready to be surprised constantly and be prepared for dangerous experiences too.

Image courtesy: http://tiny.cc/rhyd6y

Advertisements

We will have to save our world

 

This kid is telling us, adults with brains, about climate change because we apparently forgot how to think and shut our eyes to see ! And then when I listen to those adults denying climate change, my jaw drops; specially those big fat ones in the parliaments of various countries, from even specifically including the self-proclaimed world police withdrawing from the United Nations Framework Convention on Climate Change. Most parents are raising their kids; most adults are basically hating their jobs, in coming days probably won’t even find jobs; some adults are trying to invent every form of entertainment so that they can escape the reality – it seems like everybody already deluded each other believing that we are going to live in the same world that we were born somehow. We are living in the fourth but the biggest technological revolution- the internet, the social media, the money revolution, the artificial intelligence, mars exploration in one hand, on the other hand the decimation and destruction of our own planet and atmosphere by our own greed. Parents have no idea that the schools, colleges they send their kids probably won’t help their kids find any jobs as the way Google, Amazon are developing Artificial Intelligence and the automation is taking away jobs from human hands and our schools have no clue about future but to teach the same books they taught our grandparents, and then on our lands on which we build our homes, probably the most habitable lands, will no longer be available in most areas due to natural hazards resulted by climate change. Countries are still trying to steal oil from each other, fossil fuel industries are still lobbying so that they can make some extra dollars by going against green renewable alternatives, rainforests are being destroyed so that the overpopulated earth can eat more and more; but then when the cyclones and tsunamis will rise from the ocean, when the earthquakes will just decimate acres of lands, displace millions of people and force them to migrate and cause wars and battles as our vague nationalism and capitalistic self-cyclic greed are just making us more selfish and self-righteous and less compassionate day by day; when the melted ice and glaciers and the consequent sea level will just level everything within few decades; I don’t know where will all these entertainment politics and mudslinging and worrying about the smallest little things, fighting over the silliest topic on the outdated television channels will go. To be pessimistic, may be, the nature will just blow up the earth with all our interesting circus inside; just like a cruel boy uses his cruel lazy feet to destroy an ant colony that those small, little, hard working ants have built little by little, after sweating for days and nights. To be optimistic, I guess, we will have like some child genius activist like this school kid who will motivate all of us to do the right thing, to take the actions necessary, to start loving our planet, to enjoy living together peacefully with all the other species. I don’t want to live with horrors, rather I want to live in a world full of optimism and energy and with the sense that I am working towards making the world better than the world I was born. I was not aware before because I didn’t have the information, I didn’t know how our world is shaping into a nightmare because of climate change. This little kid talks about how she doesn’t want to be a climate scientist to discover another issues for the future that will not exist because of what we are doing now and now and now. We can change and we will have to change. She shows how we can change, how we can take actions to save our mother earth for our own selves. And it starts by understanding what we have done wrong, and how we can do right- like may be, we should reduce burning fossil fuels, may be we should not elect politicians who are ignorant about climate change, artificial intelligence, may we should start using electric cars to try to reduce the emission of carbon dioxide, may be we should start using renewable energy sources, so that the new renewable energy companies can florish, may be we should be vocal against industrial farming so that the greenlands, forests, rainforests don’t get destroyed, may we should take care of our ecological niches, may be we should go vegetarian or vegan so that we consume less animal products, which will reduce industrial farming which is a significant cause behind greenhouse gas emission, may be we should rethink about other kind of jobs which can employ people who are dependent on the jobs that again are destructive towards our planet. We should live in the world of today and enjoy it in a way so that the world can florish. May be, we should learn to love our world. Please educate yourself about climate change.

The volatile Bitcoin

Screen Shot 2018-06-14 at 2.04.11 AM
Technical Analysis of last 300 days of Bitcoin price data.
 
The top rectangangular frame shows the bar chart(green and orange bars) of the bitcoin price. A bar chart represents the OHLC(open, high, low, close) price for a time period (in this case 1 day) for an asset/stock/cryptocoin.
 
Bollinger Band: You can see a band (the dotted red lines above and below and a grey line in between) around the bitcoin OHLC price. The grey line represents the average price of 20 days. When the bars are above the grey line, the price is being appreciated, when the bars are below the grey line, the price is being depreciated.
 
Relative Strength Index(RSI): The blue line in second rectangular frame below shows RSI. Relative strength index(RSI) represents when the bitcoin has been overbought (more bought than being sold –> leads to price hike) or when the bitcoin has been oversold (more sold than being bought –> leads to price drop). When the blue RSI line is abobe 80 (check in the left y axis), it’s normally overbought (price rise) and you can see direct correlation of price hike comparing with the price chart above, when the blue RSI line is below 20, it’s normally oversold (price drop). You can see, right now everyone is panicking and everyone is selling, nobody almost is buying, so price is dropping rapidly. Bitcoin has hit the lowest RSI value of the year.
 
Moving Average Convergence Divergence(MACD): The third rectangular frame below shows MACD value with grey bars, grey dotted line and a red dotted line. The red dotted line is the average price of 26 days and the grey dotted line is the average price of 12 days. When the grey line is above the red line, it means that the short term price is higher than the long term price, but when it is below, it means that current price is being depreciated from the long term accepted price. MACD is another momentum oscillator which is similar to RSI. You can directly see how it is similar by comparing them. It’s probably easier to read the MACD chart than the RSI. Because there’s the zero line (in RSI it probably represents the value 50) when the grey bars are above(upwards) zero line, it means price appreciation, versus when the grey bars are below zero line(downwards), it means price depreciation. As you can see the price is highly depreciated now.
 
Commodity Channel Index (CCI): The fourth rectangular frame below shows CCI value with red line and filled red polygons. If in between 100 and -100, the price is not taking any huge turn (bull ride or bear drop). The filled polygons above 100 show the extreme price hike, where as the polygons below -100 show the recent extreme price drop. Because of these polygons, it’s very easy to see the price movement with CCI index.
 
Hope you enjoyed learning some of the common techniques used to understand market price.
 
Chart plotted by Abdullah Khan Zehady using R (quantmod, TTR library) language. Please contact if you want to reuse for your own purpose.
 
If you think it was helpful and you learnt valuable information which can help you guide your investment, you can contribute my effort and encourage me by sending small gift via Bitcoin or Ethereum.
BTC Address: 1HXSMFpN1C6ruAutwRNQoR4uzqPLuMHzi1
ETH Address: 0x22bC6a74e5B29167D8AE40197193762A9AC60F81

Methods in Fourier Spectral Analysis

Fourier Transform

It was a significant discovery in mathematics that any function can be expanded as a sum of harmonic functions (sines and cosines) and the resulting expression is known as Fourier series. A harmonic of repeating signals such as sunusoidal wave is a wave with a frequency that is a positive integer multiple of the frequency of the original wave, known as the fundamental frequency. The original wave is called the first harmonic, the following harmonics are known as higher harmonics. Any function can also be expanded in terms of polynomials and the resulting expression is known as Taylor series. If the underlying forces are harmonic and there possibly exists some periodicity, then the use of harmonic series is more useful than using polynomials as it produces simpler equations. It is possible to discover a few dominating terms from such series expansion which may help identify the known natural forces with the same period.
Let the symbol h(t) represent a continuous function of time. The Fourier transform is a function of
frequency f.

H_T(f) = \int_{-\infty}^{\infty} h(t) e^{2 \pi i f t} dt
e^{2 \pi i f t} = cos(2\pi f t) + i sin(2\pi f t)

The amplitude and the phases of the sine waves can be found from this result. Given data h(t), we can find the Fourier transform H(f) using Inverse Fourier transform.

h(t) = \int_{-\infty}^{\infty} H_T(f) e^{-2 \pi i f t} dt

The spectral power P is defined as the square of the Fourier amplitude:

P_T(f) = |H_T(f)|^2

However, real data does not span infinite time and most likely be sampled only at a few discrete points over time. Suppose that, we received values of h(t) at times t_j, then an estimate of the Fourier transform is made by using summation. The inverse transform is also shown using the summation.

h_j \equiv h(t_j)
H(f) \equiv \sum_{j=0}^{N-1} h_j e^{2 \pi i f t_j}

The data are desired to be sampled from equally spaced time as nice statistical properties are available in such regular case. If the interval between equally spaced data points is \Delta t, then the highest frequency that will appear in the fourier transform is given by the Nyquist-Shannon sampling theorem. The theorem states “If a function f(t) contains no frequencies higher than f Hz, then it is completely determined by giving its ordinates at a series of points spaced \frac{1}{2f} seconds apart”. Therefore, the Nyquist frequency (highest frequency) is given by the following equation.

f_N = \frac{1}{2\Delta t}

The lowest frequency is the one that gives one full cycle in the time interval T. The other frequencies to evaluate is the multiples (f_k) of the low frequency f_L. And, also we can derive the symmetric pair of equations. Moreover, if h(t) is band-limited (no frequencies below f_L or above f_N), then there is a relationship between the continuous function h(t) and the discrete values H_k.

f_L = \frac{1}{T}
f_k = kf_L
H_k \equiv \sum_{j=0}^{N-1} h_j e^{2 \pi i jkf_L t_j}
h_j = \sum_{k=0}^{N-1} H_k e^{2 \pi i jkf_L t_j}
h(t) = \sum_{k=0}^{N-1} H_k e^{2 \pi i kf_L t} (when band limited)

Periodogram

Fourier transform give us the complex numbers and the square of the absolute value of these numbers represent the periodogram. This is the first form of numerical spectral analysis and is used to estimate spectral power. Even though the data points collected are at evenly spaced specific discrete time, it is possible to evaluate periodogram at any frequencies.

Fast Fourier Transform (FFT)

We can calculate the Fourier transform very efficiently by using FFT. It requires data at equally-spaced time points, and is most efficient when the number of points is an exact power of two. Interpolation is often used to produce the evenly-spaced data which may introduce additional bias and systematic eror. For real data consisting of N data points y_j, each taken at time t_j, the power spectrum outputs a set of N+1 data points. The first and the last data points are the same, and they represent the power at frequency zero. The second through to the N/2 + 1 data points represent the power at evenly-spaced frequencies up to the Nyquist frequency. The spectral power for a given frequency is distributed over several frequency bins, therefore an optimum determination of the power requires combining these information and proper investigation of leakage. FFT, generally, calculates the amplitude for a set of frequencies. N/2 complex amplitudes are calculated at N/2 different frequencies. Because, these may not be the true frequencies present in the record, we subtract the mean from the data and then pad it with zeros to overcome this challenge.

Aliasing

The time series consists of measurements made at a discrete, equally spaced, set of times on some phenomenon that is actually evolving continuously, or at least on a much finer time scale. For example, samples of Greenland Ice represent the temperature every 100 years, but if the sampling is not precisely spaced by a year, we will sometimes measure winter ice, and other times measure summer ice. Even without the existence of long-term variation in the temperature, fluctuations (jumping up and down) in the data can be noticed. So, there can be frequencies higher than the Nyquist frequency associated with the sampling interval. Thus a peak in the true spectrum at a frequncy beyond the Nyquist frequency may be strong enough to be seen(aliased) in the spectrum which may give the impression that a frequency is significant when it is not. Or, a peak may partly obscure another frequency of interest. This phenomenon is known as aliasing.

Tapering

Fourier transform is defined for a function on a finite interval and the function needs to be periodic. But with the real data set, this requirment is not met as the data end suddenly at t=0 and t=T and can have discontinuities. This discontinuity introduces distortions (known as Gibbs phenomenon) in fourier transform and generates false high frequency in the spectrum. Tapering (using data window) is used to reduce these artificial presence. The data y=f(t) is multiplied by a taper function g(t) which is a simple, slowly varying function, often going towards zero
near the edges. Some of the popular tapers are:

1. Sine taper g(t) = sin(\pi t/T)
2. Hanning (offset cosine) taper g(t) = \frac{1}{2}(1-cos(2\pi t/T))
3. Hamming taper g(t) = 0.54 - 0.46cos(2 \pi t/T)
4. Parzen or Bartlett (triangle) window g(t) = 1 - (t - T/2)/(T/2)
5. Welch (parabolic) window g(t) = 1 - (t - T/2)^2/(T/2)^2
6. Daniell (untapered or rectangular) window g(t) = 1

The frequency resolution in the spectrum of the tapered data is degraded. If the primary interest is the resolution of peaks, then the untapered periodogram is superior. However, tapering significantly reduces the sidelobes and also the bias applied to other nearby peaks by the sidelobes of a strong peak. Because, the taper functions are broad and slowly varying and their fourier transform FT(g) are narrow. The effect of tapering the data is to convolve the fourier transform of the data with the narrow fourier transform of the taper function which amounts to smoothing the spectral values.

FT(fg) = FT(f) * FT(g)

<

p style=”text-align:justify;”>
// Sine taper
t <- seq(0,1, by=0.01)
T <- 1
g <- sin(pi * t * T)
plot(t, g, t='l', col=1, ylab='g(t)')

// Hanning (offset cosine) taper
g2 <- 1/2 * (1-cos(2*pi*t/T))
lines(t, g2, t=’l’, col=2)

// Hamming
g3 <- 0.54 – 0.46 * cos(2*pi*t/T)
lines(t, g3, t=’l’, col=3)

// Parzen or Bartlett (triangle) window
g4 0.5, 1 – (t-T/2)/(T/2), 2*t)
lines(t, g4, t=’l’, col=4)

// Welch (parabolic) window
g5 <- 1 – (t-T/2)^2/(T/2)^2
lines(t, g5, t=’l’, col=5)

// Daniell window
g6 <- rep(0.5, length(t))
g6 <- ifelse(t <= 0.2, 0, g6)
g6 = 0.8, 0, g6)
lines(t, g6, t=’l’, col=6)

legnd = c(‘Sine’, ‘Hanning’, ‘Hamming’, ‘Bartlett’, ‘Welch’, ‘Daniell(20%)’)
legend(‘topleft’, legend=legnd ,col=1:6, lty=1, cex=0.75)

Multitaper Analysis

We apply taper or data window to reduce the side lobes of the spectral lines. Basically we want to minimize the leakage of power from the strong peaks to other frequencies. In multitaper method, several different tapers are applied to the data and the resulting powers then averaged. Each data taper is multiplied element-wise by the signal to provide a windowed trial from which one estimates the power at each component frequency. As each taper is pairwise orthogonal to all other tapers, the windowed signals provide statistically independent estimates of the underlying spectrum. The final spectrum is obtained by averaging over all the tapered spectra. D. Thomson chose the Slepian or discrete prolate spheroidal sequences as tapers since these vectors are mutually orthogonal and possess desirable spectral concentration properties. Multitaper method can suppress sidelobes but have higher resolution. If we use few tapers, the resolution won’t be degraded, but then sidelobe reduction won’t happen much. So, there is a trade-off which is often misunderstood.

Blackman-Tuckey Method

 

Blackman and Tuckey prescribed some techniques to analyze a continuous spectrum that was biased by the presence of sidelobes of strong peaks in the ordinary periodogram. Blackman-Tuckey(BT) method was developed before 1958, prior to the FFT(Fast Fourier Transform) method. A discrete fourier transform of N points would
require the calculation of N^2 sines and cosines. With the slower computer in the pre-FFT days, the calculation of fourier transform was thus expensive. BT method has reduced the time by reducing the size of the dataset by a factor of the lag in the autocorrelation calculation. BT method is based on a fundamental theorem of Fourier transform that the Fourier transform of a correlation is equal to the product of the Fourier transforms. The correlation of two functions g(t) and h(t) is given by the first equation below.

C(\tau)=g\otimes h= \int_{-\infty}^{\infty} g(t) h(t+\tau) d\tau
FT(g\otimes h) = FT(g) FT(h)

When g = h, it is called Wiener-Khintchine theorem. Here, P is the spectral power.

FT(g\otimes g) = |FT(g)|^2 = P

The algorithm in BT method calculates partial autocorrelation function, defined by

A_{BT}(\tau) = \int_{0}^{N/l} f(t+\tau)f(t) dt

Here, N is the length of the data set but we integrate only upto N/l. $l$ is associated with the lag. When l=3 (recommended by Blackman and Tuckey) is used, we say that “a lag of 1/3” is used. Now the fourier transform of partial autocorrelation function A_{BT} gives us the spectral power. Moreover, the symmetric property of the partial autocorrelation function (A(-\tau) = A(\tau)) saves half of the computation time.

FT(A_{BT}) = \int_{-\infty}^{\infty} e^{2\pi i ft } A_{BT}(\tau) d \tau = P_{BT}(f)
P_{BT}(f) = 2 \int_{0}^{\infty} cos(2\pi f) A_{BT}(\tau)

If l=1, then it is basically the full autocorrelation function A(\tau) and gives the same answer as the ordinary periodogram.

P(f) = 2 \int_{0}^{\infty} cos(2\pi f) A(\tau) = FT(A)

Because we are using partial correlation function instead of the full correlation, the spectral power function gets smoother. Therefore, we lose resolution in the BT method. However, it averages the sidelobes into the main peak, and thereby gives a better estimate of the true power. The smoothing in BT method is different from the smoothing when we use a taper. With a taper, the fourier transform is smoothed, where as with Blackman-Tukey, it is the spectral power which gets smoothed. A spectral amplitude that is rapidly varying will be averaged to zero with a taper. But in BT method, a rapidly varying amplitude does not necessarily average to zero, since the process of squaring can make the function positive over the region of smoothing. The tapering does not
average the sidelobes into the main peak. Because, shift in the time scale behaves like phase modulation. The sidelobes, when tapering is applied, will not have the same phase, and if averaged in amplitude, they can reduce the strength of the peaks. A major challenge in the BT method is that we will have to estimate the proper lag to use before doing all the calculations. Blackman and Tukey recommended starting with the value 1/3 for the lag.

Lomb-Scargle Periodogram

 

The classic periodogram requires evenly spaced data, but we frequently encounter with unevenly spaced data in paleoclimatic research. Lomb and Scargle showed that if the cosine and sine coefficients are normalized separately, then the classic periodogram can be used with unevenly spaced data. If we have a data set (t_k, y_k), we first calculate the mean and variance:

\bar{y} = \frac{1}{N} \sum_{k=1}^{N}y_k
\sigma^2 = \frac{1}{N-1} \sum_{k=1}^{N}[y_k - \bar{y}]^2

For every frequency f, a time constant \tau is defined by

\tau = \frac{ \sum_{k=1}^{N}sin(4\pi f t_k)}{\sum_{k=1}^{N}cos(4\pi f t_k)}

Then the Lomb-Scargle periodogram of the spectral power P(f) at frequency f is given by

$P(f) = \frac{1}{2\sigma^2}\frac{ \sum_{k=1}^{N}(y_k – \bar{y} ) [cos(2\pi f (t_k-\tau))]^2}{\sum_{k=1}^{N}cos^2(2\pi f (t_k-\tau))} +
\frac{ \sum_{k=1}^{N}(y_k – \bar{y} ) [sin(2\pi f (t_k-\tau))]^2}{\sum_{k=1}^{N}cos^2(2\pi f (t_k-\tau))}$

With evenly spaced data, two signals of different frequencies can have identical values which is known as Aliasing. That is why the classic periodogram is usually shown with the frequency range from 0 to 0.5, as the rest is a mirrored version. But with Lomb-Scargle periodogram, the aliasing effect can be significantly reduced.

Maximum Likelihood Analysis

In maximum likelihood method, we adjust the parameter of the model and ultimately find the parameters with which our model have the maximum probability/likelihood of generating the data. To estimate the spectral power, we first select a false alarm probability and calculate the normalized periodogram. We identify the maximum peak and test it against the false alarm probability. If the maximum peak meets the false alarm test, we determine the amplitude and phase of the sinusoid representing the peak. Then we subtract the sinusoidal curve from the data which also removes the annoying sidelobes associated with that peak. After peak removal, the variance in the total record is also reduced. Now, with the new subtracted data, we continue finding the other stronger peaks following the same procedure. We stop when a peak does not meet the false alarm test. We need to carefully choose the false alarm probability, as if it is too low, we can miss some significant peaks; it is too low, we can mislabel noise as peaks.

Maximum Entropy Method

It is assumed that the true power spectrum can be approximated by an equation which has a power series. This method finds the spectrum which is closest to white noise (has the maximum randomness or “entropy”) while still having an autocorrelation function that agrees with the measured values – in the range for which there are measured values. It yields narrower spectral lines. This method is suitable for relatively smooth spectra. With noisy input functions, if very high order is chosen, there may occur spurious peaks. This method should be used in conjuction with other conservative methods, like periodograms, to choose the correct model order and to avoid getting false peaks.

Cross Spectrum and Coherency

If a climate proxy a(t) is influenced or dominated by a driving force b(t), we can use cross spectrum to see if their amplitudes are similar. Cross spectrum is given by the product of the fourier transform.

C(f) = A(f) B^*(f)

where A is the Fourier transform of a and B is the complex conjugate of the fourier transform of b. If we want to know whether two signals are in phase with each other, regardless of amplitude, then we can take the cross spectrum, square it, and divide by the spectral powers of individual signals using the following equation for coherency. Coherency measures only the phase relationship and is not sensitive to amplitude which is a big drawback.

c(f) = \frac{|C(f)|^2}{P_a(f) P_b(f)}

Coherency is valuable if two signals that are varying in time, stay in phase over a band of frequencies instead of a single frequency. Therefore, a band of adjacent frequancies are used in the averaging process to compute coherency:

coherency(f) = \gamma^2(f) = \frac{|<C(f)>|^2}{<P_a(f)> <P_b(f)>}

Bispectra

In bispectra, coherency relationship between several frequencies are used. A bispectrum shows a peak whenever (1) three frequencies f_1, f_2 and f_3 are present in the data such that $f_1 + f_2 = f_3$ and (2) the phase relationship between the three frequencies is coherent for at least a short averaging time for a band near these frequencies. If the nonlinear processes in driving force (e.g. eccentricity or inclination of the orbit of earth) has coherent frequency triplets, then the response (i.e. climate) is likely to contain same frequency triplet. For example, \delta ^{18}O is driven by eccentricity, we should be able to find eccentricity triplet. Thus, by comparing the bispectrum plot of climate proxy with the bispectrum plot of the driving forces, we can verify the influences of driving forces.

## Monte Carlo Simulation of Background
Monte carlo simulation is extremely useful to answer the questions like whether the data is properly tuned or not, whether the timescale is incorrect, whether some spectral power is being leaked to adjacent frequencies, whether the peak has real structure and also to understand the structures near the base of the peak (a shoulder) in a spectral analysis. Generally monte carlo simulation is run multiple times. For each simulation, a real signal(sinusoidal wave) is generated, then random background signal is added, then the spectral power is calculated to look for shoulders. In this way, the frequency of the shoulder occurence can be measured and the randomness can be realized. It is important to create background that behaves similarly to the background in real data. Dissimilar background will cause false conclusion. We also need to estimate the statistical significance of the peaks very carefully.

(This article is a quick review of the fourier spectral analysis from the book “Ice Ages And Astronomical Causes- (Data, Spectral Analysis and Mechanics) by Richard A. Muller and Gordon J. MacDonald

Statistics and Data Exploration: Quantiles, probability distribution, Box plot and Q-Q (Quantile-Quantile) plot

Statistics and Data Exploration: Quantiles, probability distribution, Box plot and Q-Q (Quantile-Quantile) plot

Quantiles

What are quantiles in statistics?

If the data is sorted from small to big, Quantiles are the points which divide the data/samples into equal sized, adjacent subgroups. Every data sample has maximum value, minimum value, median value(the middle value after you sort the data). The middle value in the sorted data is the 50% quantile because half of the data are below that point and half above that point. A 25% quantile is the cut point in the data where 1/4 -th of the data is below the point. IQR is inter-quartile range which contains half of the data which contains the median and are higher than the 25% low-value data point but less than the 25% high-value data point.

Box Plot

A box-plot can be a good representation to show the quantiles. Box plot can take different shapes depending on the data. Here is an example:

Screen Shot 2018-04-16 at 10.05.34 AM

(image source: www.physics.csbsju.edu/stats/box2.html)

Example of Discrete/Continous Probability Distribution

In the figure below, you can see different frequency distribution. The blue data samples have most of it’s data near (0,1) interval, it’s left skewed. Check how the blue box is shifted to the left. The green data samples are normally distributed, meaning most of the data points are centered around zero. It also looks balanced. We find normal distribution in nature and in biological and social phenomena very often. The orange one shows almost a uniform distribution, where the data is spreaded across the range. And lastly a right skewed data. These are all discrete data points with discrete probability distribution. There are also very well known continuous probability distribution with continuous probability density function(https://en.wikipedia.org/wiki/List_of_probability_distributions#Continuous_distributions).

Screen Shot 2018-04-16 at 10.06.55 AM

  (Image source: https://www.otexts.org/node/620)

Below we can see the quantiles for the normal distribution- the cut points which divide the continuous range of points in equal probability area. The area over an interval (in x axis) under a continuous probability density function (like the normal distribution function below) represents the probability of the data falling into that range. In this case, the IQR is the blue box; data point in that interval has 50% probability of occurrence.

Screen Shot 2018-04-16 at 9.57.22 AM

Q-Q plot

We can use Q-Q plot to graphically compare two probability distributions. Q-Q plot stands for Quantile vs. Quantile plot. In Q-Q plotting, we basically compute the probabilities assuming a certain distribution (e.g. normal, gamma or poisson distribution) from the data and then compare it with theoritical quantiles. The steps used in Q-Q plotting is:

  1. Sort the data points from small to large
  2. For n data points, find n equally spaced points which serve as the probability using \frac{k}{n+1} where k=1, 2, ..., n
  3. Look at the data points, possibly plot it and assume the underlying probability distributions. Using the probabilities from the step 2, now you can calculate quantiles. Like in R language, you can use the quantile functions like qnorm or qgamma or qunif from the stats package.
  4. Now plot by putting the calculated quantiles in step 3 in x axis and putting the sorted data points in the y-axis. If the data points stay close to the y=x line, that means your assumption of the probability distribution was correct.

Below you can see one example, where the normal distribution is assumed for the ozone data. T

Screen Shot 2018-04-16 at 10.35.57 AM

Now you can see the gamma distribution fits better to the ozone data than the normal distribution.

Screen Shot 2018-04-16 at 10.37.45 AM

This is how you can check different probability distribution for your data using simple Q-Q plot. There is a fantastic Q-Q plot tutorial from which I collected the above image. For further reading, please check https://www.r-bloggers.com/exploratory-data-analysis-quantile-quantile-plots-for-new-yorks-ozone-pollution-data/ and https://www.r-bloggers.com/exploratory-data-analysis-quantile-quantile-plots-for-new-yorks-ozone-pollution-data/

 

How do Paleoclimatologists investigate about ancient Earth? What are different Climate Proxies and what are their significance?

To know and understand about ancient climate, different climate proxies are generally used. We can measure the concentration of greenhouse gases by using entrapped air in the Greenland and Antarctic glaciers which give us samples of the atmosphere back to about 420 Kyr. The glaciers in North America and on mountains in tropical Andes can be estimated from scour marks, moraines and erratic boulders.Forams are microscopic organisms whose life cycles depend on local temperature and whose fossils preserve samples of ancient material. Some planktic forams (short for foraminifera) represent a “proxy” for sea surface temperature as they indirectly inform us about the temperature. One of the most remarkable proxies is the ratio of oxygen isotopes in benthic(bottom dwelling) forams in ancient sediment, which reflect the total amount of ice that existed on the Earth at the time the sea beds were formed. A scientist needs to be careful in their analysis as most proxies are dependent on more than one aspect of climate. Now I will discuss the primary proxies which have been used to investigate paleoclimate. Many of the samples come from seafloor cores, cores from Greenland or Anatarctic ice. The cores are named V22-174, RC13-110, DSXP-806 etc. In the geologic community, various of these prefixes are used some of which are enlisted below:

  • V: Vema, a converted yacht operated by Lamont-Doherty Earth Observatory of Columbia university.
  • RC: Research vessel Robert Conrad.
  • DSDP: Deep Sea Drilling Project operated from 1968 to 1983 by the Scripps Institution of Oceanography at University of California, San Diego.
  • ODP: Ocean Drilling Program as an international collaboration.\newline
  • GRIP: European based GReenland Ice-core Project.
  • GISP2: US-based Greenland Ice Sheet Project #2.
  • Vostok: Russian station on the East Antarctic ice plateau.
  • MD: The research vessel Marion Dufresne, operated by the French.

1. Oxygen Isotopes

The pattern of oxygen isotopes is remarkably similar in sea floor records around the world and this universality feature is very attractive for a climate proxy. The ratio of oxygen istopes found in ice, trapped air, benthic/planktic forams is widely used as a climate proxy. Oxygen consists of three stable istopes: 99.759% is ^{16}O, 0.037% is ^{17}O, and 0.204% is ^{18}O. The variation in the fraction of ^{18}Olatex can be measured with high accuracy. The fractional change, shown by the following equation, basically means that how much difference of the ratio of \frac{^{18}O}{^{16}O}latex exists in perts per thousand in the sample compared to the reference.

\delta^{18}O = \left(\frac{\left(\frac{^{18}O}{^{16}O}\right)_{Sample}}{\left(\frac{^{18}O}{^{16}O}\right)_{Reference}} - 1\right) \times 1000

 

Oxygen isotope separation occurs because of the isotopic differences in vapour pressure and chemical reaction rates, which depends on temperature. Some of the most important geophysical processes that lead to changes in \delta^{18}O are:

  1. Evaporated water is ligher than the remaining liquid. Water containing ^{16}O has higher vapor pressure than water containing ^{18}O, so it evaporates quickly.
  2. Precipitated water molecules are heavier than those in the residual vapor. H_2^{18}O condenses more readily than H_2^{16}O, so as water vapour is carried across to Greenland or to central Anatarctica, the residual becomes lighter.
  3. Oceanic \delta^{18}O in non-uniformly distributed. It means that the changes in the pattern of winds that carry vapor and change the source will also change \delta^{18}O. At present, the difference in surface water is 1.5% from pole to equator.
  4.  Biological activity enriches the heavy isotope. The \delta^{18}O in the calcium carbonate of shells is 40% greater, on average, than in the water in which the organism lives.

 

The net result of these effects is that glacial ice is light, with \delta^{18}O typically lower than seawater. So, in glacial ice containing more ^{16}O, \delta^{18}O is negative, where as in surface water containing more ^{18}O, \delta^{18}O is positive. However, when large volumes of ice are stored in ace-gage glaciers, then there can be considearable depletion of the light isotopes in the oceans.
In 1964, Dansgaard and colleageus showed that measurements of isotopic enrichment in ocean water as a function of latitude yield the following approximate relationship between temperature T and \delta^{18}O:

\delta^{18}O \equiv 0.7 T - 13.6
However, there can be other factors in the change of \delta^{18}O. Therefore, if we go back to earlier when the temperature was lower, \delta^{18}O might not be lower which contradicts the above equation. When several measurements are made at the same latitude, the effect is argued to depend on the amount of precipitation and not on temperature.
Moreover, depending on the source, we will have to consider other issues. In planktic fossils, we might expect \delta^{18}O to reflect surface conditions, and therefore be sensitive to temperature and salinity conditions. In benthic forams, \delta^{18}O must be more sensitive to global ice, since there is little temperature variation on the sea floor. In other samples (e.g. ice, trapper air or calcite), \delta^{18}O may represent the temperature, not ice volutme.
Several attempts have been made to extract the underlying \delta^{18}O signal that is common in the records. SPECMAP stack (Imbrie et al., 1984) was a combination of five $\delta^{18}O$ records from five cores: V30-40, RC11-120, V280238 and DSDP502b.

2. Deuterium – Temperature Proxy

Hydrogen generally contains only one proton in its nucleus and is lighter with atomic weight 1. Deuterim (D or ^2H), on the other hand, is one of the heavy isotopes of hydrogen which contains one proton and one neutron in its nucleus and thus the atomic weight is 2. Bonds formed with deuterium tend to be much more stable than those with light hydrogen. The deuterated water is more sensitive to temperature than that of ^{18}O. We can clearly see it in the “fractionation factor” which describes the equilibrium between liquid and vapour. The fractionation factor is defined to be the ratio of D/H in a liquid to the ratio of D/H in a vapor that is in equilibrium with that liquid. The fractionation factor for HDO is approximately 1.08 and it varies more rapidly with temperature compared to ^{18}O. Therefore, the condensation of the deuterised form of heavy water (HDO) is significantly more sensitive to temperature variation than is the ^{18}O form (H_2^{18}O). Therefore, deuterim is considered as a temperature proxy. A temperature scale was devised fro the Vostok ice core by assuming the equation:

\Delta T = \frac{\Delta \delta D_{ICE} - \Delta \delta^{18}O_{SW}}{9}
where, the $\delta^{18}O_{SW}$ refers to the sea floor isotope record.

3. Carbon-13

Carbon on the earth has two stable istopes, ^{12}C with an bundance of 98.9% and ^{13}C with an abundance of 1.1%. The ratio of these two isotopes is described by the quantity \delta^{13}C and defined by the equation below. The reference value is often taken to be a sample known as the “Peedee belemnite” (PDB); its \delta^{13}C value is very close to that of mean sea water.

\delta^{13}C = \left(\frac{\left(\frac{^{13}C}{^{12}C}\right)_{Sample}}{\left(\frac{^{13}C}{^{12}C}\right)_{Reference}} - 1\right) \times 1000

The lighter isotope, ^{12}C, is easily absorbed into the organic tissue of plants, leading to negative values for ^{13}C = -20% to -25%. In regions in which photosynthesis is active, this removes typically 10-20% of the dissolved inorganic carbon in seawater, leading to ^{13}C enrichment in surrounding water. Because different regions of the world have different activity, there is geographic variation. Warm surface water has the highest \delta^{13}C, where as deep Pacific water has the lowest \delta^{13}C. Thus \delta^{13}C can be used as a tracer for oceanic currents.
In contrast, there is only small separation of carbon istotopes that takes place in the formation of caclcium carbonate shells. Thus the measurement of \delta^{13}C reflects the composition of the ocean water at the time and location in which the shell grew.
^{13}C is extremely important isotope for paleoclimate studies, because it responds to the presence of life. \delta^{13}C can record climate change. During glacial periods, biological activity was reduced by advancing glaciers and colder temperature, and light carbon was released into the atmosphere and eventually mixed into the oceans. \delta^{13}C from benthic (bottom dwelling) forams is typically 0.35% lower during glacials than during interglacials. In contrast, planktic forms don’t show such changes.

4. Vostok

The ice core from the Vostok site in Antarctica (Petit et al., 1999) located at 78^oS and 107^oE, covers the longest period of time of any ice record. It reached a depth of 3623 metres. A untuned but unbiased timescale was derived based on ice accumulation and glacial flow models. Many proxies of climate interest were measured in the Vostok core, including atmospheric methane, atmospheric oxygen, deuterium in the ice, dust content and sea salt. Atmospheric methane is produced by the biological activity of anaerobic bacteria and it’s existence in paleoclimate data is presumed to reflect the area of the earth covered by swamps and wetlands. The observed dust (strong 100 Kyr cycle) in the Vostok dust record is beleived to reflect reduction in vegetation during those periods and accompanying increase in wind-blown erosion. Then, the sodium concentration reflects the presence of sea spray aerosols blowing over the Vostok region.

5. Atmospheric \delta^{18}O and Dole Effect

The atmospheric oxygen has a \delta^{18}O of +23.5% compared to that of mean ocean sea water due to the removal of lighter isotope ^{16}O from the atmosphere by biological activity. The difference is called the “Dole Effect” and it is assumed to be time-independent.

6. \delta^{18}O / $CO_2$ Mystery

The difference between ocean and atmospheric $\delta^{18}O$ is due to the biological activity. However, carbon dioxide, even though driven by biological processes, doesn’t show similar spectra. The strong peaks in the oxygen signal forced by precession parameter is absent in the carbon dioxide record which is mysterious and still under investigation.

7. Other Sea Floor Records

7.1 Terrigenous component

The terrigenous component of sea floor sediment is the fraction which has possibly come from land, in the form of wind-blown dust. The most significant frequencies which have been found in the spectrum of detuned terrigenous component Site 721 are marked with the periods: 41, 24, 22 and 19 Kyr. These periods indicate that the signals were dominated by solar insolation.

7.2 Foram size: the coarse, or “sand”, component

In the sea floor core, the main component of the sand is frequently large forams. Therefore, the coarse component reflects an interesting change in the ecology of the oceans. A clear eccentricity signal was detected in a core that already showed a clear absence of eccentricity in the \delta^{18}O component.

7.3 Lysocline: carbonate isopleths

Pressure varies in different depths of the ocean and which consequently influences the solubility of the calcium carbonate. At a certain depth, the shells of fossil plankton begin to dissolve, and this boundary is called lysocline. It can be quantified by the percentage of calcium carbonate in the sediment, as a function of depth. One can plot the depth at which the 60% lysocline is found, as a function of age and this depends on the depth of the oceans at that age. The signal apeears to be dominated by a 100 Kyr cycle, as would be expected if the primary driving force were the depth of the ocean, determined by the amount of ice accumulated on land.

Featured Image Courtesy: https://katiecoleborn.wordpress.com/5-proxy-climate-records-what-are-they-and-how-do-they-work/
Main Reference: Ice Ages and Astronomical Causes (Data, Spectral Analysis and Mechanisms) by Richard A. Muller and Gordon J. MacDonald.

Amazon book link:

Immigration problem

Just posting my Quora answer..
Intelligent, peaceful, skillful immigrants are needed not only in US but also in every other country. Because cultural assimilation is a dire need of our time. If you look at most of the problems we have in our society, they are born out of close mindedness of the people. Why racism is a problem, coz some people just deny or at least used to deny human rights to people who have darker skin. Why an immigrant can find job, but an American can not? You are born in America, but you haven’t explored much or expanded your horizons much to find a job. An immigrant is crossing the ocean or a desert and taking the risk and learning his way to live in a society by learning a different language. That is what a great great great grandparents of a jobless American might have done that the great great great grand son or daughter now doesn’t understand. I have met Americans in Japan when I lived there. When they couldn’t find job in America, they went to Japan and started teaching English to Japanese kids, married Japanese women and are now living there happily. My American friend now lives in Shikoku, Japan and I’m now living within 3 hours of his American home in Michigan. This is how the world should look like. We assimilate, we know, we challenge, we meet people, we make friends, we figure out what we like or dislike, we grow our potential and we live. Nobody is entitled to have anything for granted. Reading a little bit of history will give more persepective. Seeing some statistics on social well being, education, contribution to society by race, by country will open your mind. British have run the world for hundred years, where are they now? America in the sixties have had the highest and happiest middle class, what happened now? World, society will forever change and we should be resilient enough to adapt with the change. Instead of blaming immigrants taking away jobs, an American should rather unite with other Americans of all color and race and legal peaceful immigrants to see how politics and legislations in this country can be made so that people have more rights, more jobs and happy lives, how it’s possible to inspire people or communities rather than infuriating them. Yes, illegal immigration is a problem, but what is to do now to ensure the overall well being of everyone should be the focus or priority. Sometimes sentimental talks and arguments just divide us too much; rather constructive criticism and thoughts should unite us to find real solution to the real problem towards welfare. Putting people in suffering and building walls will never bring wealth and prosperity. So, we should all be critical thinker, empathetic and creative to create a great society. Let’s move our lazy ass, educate ourselves, do what needs to be done in our own space within our reach without complaining too much!

https://www.quora.com/Why-are-immigrants-needed-in-the-U-S-when-many-Americans-dont-have-jobs/answer/Zehady-Abdullah-Khan?share=f8d596b1&srid=X0U1

Fossil Record as a way to learn earth history

Who studies fossils? A Paleontologist. Why do paleontologists study fossils? Because fossil record brings information, provides clues, ideas about climatic changes of the planet, the evolution of geographical changes occurred on earth. For example, plant fossils and pollen fossils have been used to indicate climatic change of earth. To create geologic timescale, scientists have used fossils. We can see the fossil evolution during Paleogene period (over 66 Million years ago) in the image below. You can see the branches and subbranches in the image. The Subbotina trivialis (genus: Subbotina, species: trivialis) is highlighted in red.

Screen Shot 2017-07-26 at 9.19.52 PM
Evolution of Planktonic fossils during Paleogene period. The fossil image of Subbotina Trivialis has been shown in the rectangular yellow window. Subbotina Trivialis has found around 65.5 Million years ago.

Determining the age of fossil is very important and very challenging. Fossils are very often found in rocks and comparing one rock formation with another (relative dating), it’s possible to find a relative age for a fossil. Dating rocks involve calculating the rate of decay of radioactive element such as Carbon-14, Uranium-238, Potassium-40, Aluminium-26, Samarium-147, rubidiam-87, strontium-87. Fossilization is a rare event as there may not be any trace of an organism after its extinction. Therefore the record of an organism as the record of life in a fossil is something very significant to discover. The organism’s physical structure and subsequently deduced information such as it’s environment, diet, life-cycle can be obtained by studying fossils. Trace fossils, or fossilized marks left as a result of the activities of creatures such as trails, footprints, and burrows are also recorded and used as the source of information. From the fossil records throughout geologic time, scientists understood that the evolution of life is not a linear process. Sometimes the process is slow and sometimes it’s exponential. We also discovered that there might be periodicity in mass extinction by studying fossil records. Even the concept of plate tectonics was helped by fossil records. The more I am learning about fossils, the more exciting it’s becoming.

Does Ideology Matter?

I have been talking with a friend about whether ideology matters in our life? It seems “Ideology” has no role in our everyday life. But is that true? Certainly every person has his own outlook towards life and prefer to take certain stance over all other possible stances. It surely depends on how we have been taught, what school we went, what society we have been living. And our belief, attitudes toward life should definitely have some sort of relationship with the ideologies which were dominant in our environment. However, I was searching on google “Does Idelogy Matter?” And I came in touch with some interesting articles, specially a chapter from a book called “Introduction: The Debate over Ideology” by John Schwarzmantel. I got intrigued by some of the sentences by the writer in the first chapter which I will mention next:

It seeks to present the features of those ideas which move people to action in the contemporary world, and seeks to answer the question of whether we are in a post-ideological society, in which the ideas that dominated the modern world and spawned mass movements, political parties and demands for revolutionary change, have lost their appeal. Have these inspiring ideals ceased to mobilise people, and been replaced by other ideologies, of different nature and origin, with completely transformed political implications? Or is the picture a different one, in which political life, at least in ‘developed’ countries, is marked precisely by an absence of overarching ideas or ideologies, with scepticism and hostility to such broad ideologies as characteristics of our time?

He points out that the political ideologies have moved people to action in the past:

In more mundane and less dramatic forms, political life in many countries has been animated by hundreds, indeed thousands of people engaging in political activity, sometimes of a very humdrum kind, because they believed that they were making some contribution, however small, to the victory of their ‘cause’. So there seems plenty of historical evidence that politics can not be understood without comprehension of ideas or packages of political ideas that have mobilised people to political activity, at whatever level. We are talking here not just of great leaders, charis- matic orators, founders or leaders of political parties, but of masses of people who found in political ideals an inspiration and a cause.

Then he poses the doubt whether it’s also the same now?

Has the shape of the contemporary world, or its ideological configuration, shifted, so that instead of mobilisation for visions of ‘the good society’ we now live in a society focused on maintaining certain identities, and defending a group’s dignity, or respect?

He brought the suggestion of philosopher “Charles Taylor” about equal recognition of our identity:

A politics of equal recognition has come to play a bigger and bigger role’ (Taylor, 1994: 37). We want to be recognised as beings of equal dignity, which includes recognition of our particular identity, which gives us a sense of authenticity…The political life of contemporary ‘developed’ societies, and perhaps worldwide, is dominated by a struggle for recognition and respect. The overall aim is that one’s authentic culture, religion and customs are given ‘space’ and respect. This then takes priority over more ideological concerns, which are broader and more sweeping in their scope. Ideas of liberalism, socialism, conservatism, among others, offer more general aspirations, and stem from a common ‘Western’ heritage.

Then he says why in a multicultural world someone who is of different origin may not be interested and how there can be a crisis of ideologies:

…may not be appealing to those whose cultural origins lie elsewhere, who reject the proclaimed universality of those ideals of ‘Left’ and ‘Right’, and seek recognition and respect. This would account for a crisis of ide- ologies, in which the main ideologies of the Western tradition have lost their mobilising capacity. This would be, at least in part, because those ideologies have a certain cultural underpinning, operate with certain assumptions of progress, rationality, secularism and with a certain pretension to universality. These are all assumptions which have come under suspicion in a much more multicultural world which exalts difference and diversity, and which is more receptive to identity than to ideology.

He asked:

…whether in truth identity has replaced or reduced the importance of ideology, or whether new ideologies which give more importance to ‘the politics of recognition’ have superseded older ideologies which underplayed issues of cultural identity.

American philosopher Richard Rorty distinguishes between “ideological politics and issue politics”. Rorty’s argument is that:

A politics of ‘movements’ orients political action to some grand overar- ching aim. Particular issues are judged in terms of their contribution to the final goal of overall social transformation.

The implication is that the present may be sacrificed in the light of a better future. The movement is way more important than the final goal. Movement politics can equally be called ideological politics. On the other hand Rorty calls campaigns the issue politics. The definition of campaign is given in this way:

Campaigns are precisely about issues, about specific matters which are fought for in a limited way: rights of a particular group, a particular instance of environmental pollution or contamination, for instance. When people take to the streets in the societies of contemporary liberal democracy, it is on particular matters: protests against going to war in Iraq, a demonstration against tuition fees for university students, protests against reform of pension laws and welfare measures..

 Campaigns have a finite perspective. They are not seen as contributing to a future and a different society but wish for change in the present, without the aspiration to build a new society. He thinks that the ideological politics has been replaced by issue or campaign politics:

We are in a society of a ‘post-ideological’ kind, where it is not the struggle to create a better and totally different society, which occupies that section of the society, itself perhaps a minority, which engages with political activity….We have become a society focusing on issues which are remediable in the here and now.

It seems very true to me. We are now really living in a society where we don’t want everything to be changed. We just probably want improvement of situations and we extremely love to complain. However the author then declares that this is a false distinction:

Any struggle for a particular issue, or campaign, can only be justified in terms of a general philosophy, or ideology. To protest against a motorway or out-of-town supermarket is to be spurred on, maybe only implicitly, by a general ideology of ecologism or green politics. To join in a march against war in Iraq might not be necessarily consciously equated to an affiliation to socialism or any ideology of the Left, but it seems to fit in with a broader philosophy of preferring peaceful resolutions of conflicts though international organisations such as the United Nations rather than the hegemony of the United States.

Then he claims:

Issues can only be identified as such within the framework provided by more general frameworks of ideological politics which give a map of the world, metaphorically speaking, and make it possible to specify why some- thing is an issue and is worth fighting about, or demonstrating about.

He talks about totalising or totalistic ideologies and partial or fragmented ideologies. The totalistic ideology invokes a picture of overall social and political transformation. But the latter takes a less holistic view of society and social change and only tries to remedy specific grievances. And where the sacrifice of the present generation for the sake of some future goal is refused.

There are more detailed explanation along this line throughout the first chapter. And in the end, it is shown to be evident that the situation now is different:

The situation now is different and more problematic, in that while aspira- tions to community and to a better or more satisfying society remain strong, they are more difficult to realise in a society which is not only more frag- mented, but where the goals of community have to be recast at an international level….

The nation-state is now a much weaker building block, since it has been hollowed out both internally, by a variety of cultures, and externally, by flows of market forces…

…in the present situation, the forging of such sentiments of national solidarity, and of other kinds of community as well, comes up against greater obstacles or problems.

How the movements of people and commodities are bringing changes:

…the much greater movements of immigration and flows of people and commodities across national borders have created aggregations of people to whom the traditional ideologies are of diminishing relevance…

…is to assert firstly that integrative and mobilising ideologies of politics are necessary in the contem- porary world, but, secondly, that the available ones are so badly crippled by changes in real life that they (and the aspirations they articulate) have to be reformulated in ways that may make them unrecognisably different from their previous forms.

We are probably living in “beyond left and right”:

Giddens argues that the traditional antithesis of Left and Right is no longer appropriate to describe the ideological scene of contemporary liberal-democratic societies. The Left,  used to be in the vanguard of progress and modernity, standing for an egalitarian society in which the state had a crucial redistributive role, exemplified by the welfare state. Yet this idea of a directing and coordinating state, which acts in a somewhat paternalistic way, is out-of-date in a more complex society of ‘reflexive modernity’. Such a society is marked by greater individualism, a rejection of the state imposing uniform ways of satisfying people’s needs.

..the traditional Right is in equal difficulties. Appeals to traditional authority and a concept of hierarchy, both of which were central to old-style conservatism in Britain and elsewhere, havelost their relevance in a society where tradition is a much weaker force, where a more educated population rejects the deference and acceptance of estab- lished institutions on which parties of the conservative Right used to rely…

The conclusion drawn is that in a society beyond Left and Right, a new ideological approach is needed, which focuses more on ‘lifestyle politics’, issues of consumption, individualism and the pri- vate sphere which the established ideologies of the past neglected.

 However, I want to finish with his final argument:

It is argued here that this does not make ideologies redundant, but on the contrary that the impoverishment of much of political life in liberal-democratic societies stems precisely from the weakness of ideological politics – the lack of broad visions which offer a goal to be striven for.

Ideology is needed actually.