Exciting Days to come

I woke up at the middle of the night and am just thinking. Sometimes I hate when I can not stop thinking, but then sometimes thinking is fun as new ideas pop up. My friend Zak convinced me to stay in Chicago another day. It was a fun weekend. I met another of my friend too. Because Zac drives Uber in Chicago, I could see the city better than ever. The city in the evening is very colorful and dreamy. There’s a lot of museums that I’d like to visit later on. I also made some new friends. I really would love to live in Chicago if given opportunity. But then I haven’t seen so many places yet. I met Zak’s friend whom he met through his exchange program in Denmark. She is really interesting and fun to be around. But the thing that striked me is that, she is german, doing her intern in a company in Chicago,  already travelled to Mexico, now going to SF, then to Vegas, then to Costa Rica, to Mexico, to Cuba, then to Madrid, then back to Germany. Then on last Friday, I met a new friend while having dinner at an Egyptian restaurant back in West Lafayette. She has already traveled India, the next country to mine before me, and advised me to go to all these places she liked like Rajsthan, Bombay etc. I meet these people and I wonder that I haven’t done any of these. The last two years were so dark, morbid for me that I forgot to live. Anyway, I’m just going through a lot of realization how I got stuck in small thinkings, how I just always worry about what other people will think, how spiral life had become and how interesting I can make my life if I want and work for it, so many things I wanted to do, want to do, travel, I am feeling like I want to live again. I did those my entire life, I was exuberant. All that take is the desire to explore and I hate to say that all these worries, anxieties, hurt feelings just take the dreams and desires away. I still don’t know how to swim well, I want to do scuba diving, snow boarding, I wanted to learn surfing this summer but I couldn’t make it, I want to improve my Latin dancing, I want to learn spanish. After living so close from Chicago and Indiana, I didn’t know that there’s such beautiful beaches nearby. There’s just so many things near me I don’t know because I just don’t always try to reach outside, talk to the person sitting next to me. I just never want to get stuck again, feel less confident about myself, feel burdened, feel the meaninglessness of life, rather I want to create and be adventurous as I always used to be, strive to see things, learn and grow. I’m just so happy that I made this trip. And I deeply felt what it means by the words “You just can not fathom things about outside, even about yourself unless you get out of your nest.”

Advertisements

Looking Inside Eminem’s Lyrics – part 1

I started analyzing the lyrics of Eminem. My initial interest is that what are the most common words Eminem has been using in his lyrics. I collected the name of all his 287 songs from this link. Then I collected the lyrics using python sontext library which collects lyrics from lyricwiki.org. I have become successful in gathering lyrics for 223 songs of Eminem out of those 287 song names using my python code. After gathering the lyrics, I had to process the text in the lyrics. Normally in language processing tasks, we do chunking, lemmatization, stemming, spelling error check. I have used NLTK library for all these. Actually I had to avoid doing lemmatization as it was chopping off lots of interesting words in its existing form. And also I created a banned words list as Eminem has used a lot of ‘na’, ‘wa’, ‘oh’ kind of words which semantically doesn’t have much meaning.  Then I used NLTK word frequency method to find out the frequency of words. The top 20 words used were

[(u’like’, 1375), (u’get’, 1049), (u’got’, 907), (u’shit’, 740), (u”’cause”, 729),

(u’know’, 701), (u’back’, 674), (u’fuck’, 671), (u’eminem’, 593), (u’go’, 557),

(u’see’, 514), (u’one’, 497), (u’say’, 476), (u’never’, 430), (u’bitch’, 428),

(u’man’, 428), (u’let’, 422), (u’time’, 411), (u’come’, 392), (u’think’, 361)]

And yeah apparently Eminem has cursed a lot in his songs. As you can see in the plot below for the word “shit” (rank:4), “fuck” (rank:8), “bitch” (rank:15).

top_20_words
Frequency of top 20 words

Then the word ‘love’ has been used 282 times just bit less than the word ‘ass’ which was used 295 times. You can see the word ‘dre’ has been used a lot and it’s most likely Dr. Dre who worked with Eminem. The word ‘man’ is used more than the word ‘girl’. The word ‘hate’ is used less than the word ‘love’, only 116 times. Here’s two more plots for word frequency.

top_50_words
Frequency of top 50 words
top_100_words
Frequency of top 100 words

Anyway, simple bag of words probably don’t give good representation of a particular song. For example, the word love can be used in a sentence “I love you” but then also “I don’t love you” which has completely opposite meaning. But here they are counted all together. Before contextual analysis, I was just thinking about doing another frequency analysis according to Russel’s model of mood. You basically divide the xy-plane into four orthogonal regions as you can see in the image below.

Figure-1-Four-basic-mood-categories-based-on-the-PANAS-model-by-Watson-and-Tellegen
Model of mood

I want to see where eminem’s music in general fall in this emotional plane. There’s more interesting analysis I can do later on using word vector and other new NLP techniques. I’ll eventually look into other artists, other genres and try to find whether there are different patterns in how the words are chosen and the kind of emotion certain songs may generate.

Code for getting Lyrics:

import lxml
from lxml import html
import requests

import pickle

import numpy as np
import libsongtext
from libsongtext import lyricwiki
from libsongtext import utils

import pprint as pp

artist_name = 'eminem'

url = 'http://www.spin.com/2014/10/eminem-every-song-ranked/'
#f = urllib.urlopen(url)
f = requests.get(url)

html_page = f.content#f.read()
tree = html.fromstring(html_page)

song_name_xpath=tree.xpath('//div[@class="article-content clearfix"]//strong/a')
song_names=[]

num = 1
lyrics_list = {}
lyrics_not_found_list = []
success_lyrics_cnt = 0
for s in song_name_xpath:
song = ''.join(s.text.encode('ascii', 'ignore').strip())
song_names.append(song)

print 'No. ' + str(num)
num = num + 1
print 'track : ' + song

try:
args = {}
args['artist'] = artist_name
args['title'] = song.strip()
title = args['title']
if not lyrics_list.get(title):
t = lyricwiki.LyricWikiSong(args)
lyrics = t.get_lyrics()
print "Got Lyrics."
lyrics_list[title] = lyrics
success_lyrics_cnt += 1
except:
lyrics_not_found_list.append(song)
print "Failed to get Lyrics."

print 'Successfully got ', success_lyrics_cnt, ' lyrics out of ', len(song_name_xpath), ' tracks'

def save_obj(obj, path, name):
with open(path + name + '.pkl', 'wb') as f:
pickle.dump(obj, f, pickle.HIGHEST_PROTOCOL)

def load_obj(path, name):
with open(path + name + '.pkl', 'rb') as f:
return picle.load(f)

save_obj(lyrics_list, '/Users/andy/Documents/projects/music/lyrics_database/eminem/', 'eminem_song_lyrics')

Code for word frequency analysis in Lyrics:

import pickle
import string

import nltk
from nltk.tokenize import sent_tokenize
from nltk import word_tokenize
from nltk import sent_tokenize
from nltk.corpus import stopwords

import enchant
from enchant.checker import SpellChecker

eng_dict = enchant.Dict("en_US")

#import lyrics of eminem
eminem_lyrics_pickle_file='/Users/andy/Documents/projects/music/lyrics_database/eminem/eminem_song_lyrics.pkl'
f = open(eminem_lyrics_pickle_file, 'rb')
lyrics_list=pickle.load(f)

song_names=lyrics_list.keys()
lyrics= lyrics_list.values()

# english words
#words = set(nltk.corpus.words.words())

#stemmer
porter = nltk.PorterStemmer()
wnl = nltk.WordNetLemmatizer()

def plot_freqdist_freq(fd,
max_num=None,
cumulative=False,
title='Frequency plot',
linewidth=2):
"""
As of NLTK version 3.2.1, FreqDist.plot() plots the counts
and has no kwarg for normalising to frequency.
Work this around here.

INPUT:
- the FreqDist object
- max_num: if specified, only plot up to this number of items
(they are already sorted descending by the FreqDist)
- cumulative: bool (defaults to False)
- title: the title to give the plot
- linewidth: the width of line to use (defaults to 2)
OUTPUT: plot the freq and return None.
"""

tmp = fd.copy()
norm = fd.N()
for key in tmp.keys():
tmp[key] = float(fd[key]) / norm

if max_num:
tmp.plot(max_num, cumulative=cumulative,
title=title, linewidth=linewidth)
else:
tmp.plot(cumulative=cumulative,
title=title,
linewidth=linewidth)

return

stem_tokens = ['ed', 'ies', '\'s' , 'n\'t', '\'m', '--', '\'\'']
banned_words = ['ha', 'wa', 'ta', 'u', 'i', 'ai', 'na', 'ca', '...', '..', '\'em', '\'en', 'wan', '`', '``',
'oh', 're', '\'re', '\'ne', 'yea', 'yeah', 'ya', 'yah', '\'ve', '\'d', 'wo', 'oh', 'ooh',
'\'ll', 'yo', 'is\\u2026', 'ah', 'wit', 'would', '\\u2019']

#['i\'ma', 'y\'ll']

def synonyms(word):
syns = []
for word in wn.synsets(word):
sim_words = word.similar_tos()
sim_words += word.lemma_names()
for sim in sim_words:
s = sim
if hasattr(s, '_name') :
s = sim._name.split(".")[0]
syns.append(s)

syns = set(syns)
return syns

def stem(word):
for suffix in stem_tokens:
if word in banned_words:
return False

if word == 'suffix' or word.endswith(suffix):
return word[:-len(suffix)]
return word

lyrics_edited = []
chkr = SpellChecker("en_US")

edited_tokens = []
i = 1
for s, l in lyrics_list.items():
print i, ". Processing song: \"", s, "\""
i += 1
# find wrongly spelled words
chkr.set_text(l)
err_words=[]
for err in chkr:
err_words.append(err.word)

#tokenize
tokens = word_tokenize(l)
l_txt = nltk.Text(tokens)

for t in tokens:
tn = t.lower()
#tn = porter.stem(t)
#tn = wnl.lemmatize(tn)

tn = stem(tn)
if tn and tn not in err_words and tn not in stopwords.words('english') and tn not in list(string.punctuation):
edited_tokens.append(tn)

uniq_tokens = set(edited_tokens)

fdist = nltk.FreqDist(edited_tokens)

#Rusell's Model of mood
mood_happy_words = ['Exhilarated', 'Excited', 'Happy', 'Pleasure']
mood_h = []
for ws in mood_happy_words:
for w in synonyms(ws):
mood_h.append(w)

mood_h = list(set(mood_h))

mood_angry_words = ['Anxious', 'Angry', 'Terrified', 'Disgusted']
mood_a = []
for ws in mood_angry_words:
for w in synonyms(ws):
mood_a.append(w)

mood_a = list(set(mood_a))

mood_sad_words = ['Sad', 'Despairing', 'Depressed', 'Bored']
mood_s = []
for ws in mood_sad_words:
for w in synonyms(ws):
mood_s.append(w)

mood_s = list(set(mood_a))

mood_relaxed_words = ['Relaxed', 'Serene', 'Tranquil', 'Calm']
mood_r = []
for ws in mood_relaxed_words:
for w in synonyms(ws):
mood_r.append(w)

mood_r = list(set(mood_r))

Feeling Home

It’s nice to be home. A place where it feels comfortable, where you feel to come back, where you can relax, where you can leave the worries away and just chill. I just moved to a new apartment, actually quite close to my last one. I hate the arduous process of moving, but I’m happy to move to this new place. I like how the surrounding is calm and quieter, love the greens and the shades from the tree. The small front lawn up to the back door through the living room and the kitchen look like a wide corridor for the wind and light to pass through. I actually was excitingly decorating my living room so that it doesn’t look empty and feel bit artsy. I went to the Walmart and bought some cheap table decoration stuff. My favorite one is the hour glass. I was lucky to meet this family in my town who were moving to a new house and they cheaply sold a lot of furniture to me which are mainly occupying my living room now. I have a decent music system in the living room but I wish I had better speakers. I wored several hours on my wall posters, I was originally thinking about buying posters but good posters are so expensive and I couldn’t find the kind of posters I like. I wanted to have some inspirational quotes and good expressions and I wanted it to be cheap. So, in the end, I found some large nice google images with the kind of quote, characters, expressions, then I pasted them on excel and printed partial pieces on A4 size papers on which then I had to cut edges for several hours and pasted them on the wall according to the order. Now I have a Robin Williams Poster which says,

I say to you, ‘Hey, look at me.’ Please, don’t worry so much, because in the end none of us have very long on this earth – life is fleeting. And if you’re ever distressed, cast your eyes to the summer sky when the stars are strung across the velvetly night, and when a shooting star streaks through the blackness turning night into day, make a wish and think of me. Make your life spectacular. I know I did.

Then at its left, above my guitar, I have the phenomenal Jimmy Hendrick’s poster,

When the power of love overcomes the love of power, the world will know peace.

And my most favorite one in the opposite wall is from Einstein, 

Gravitation is not responsible for people falling in love.

 Anyway, I am happy how it looks and feels now. And I think my friends liked my place too. Yesterday I had my housewarming party. A lot of my friends and their friends came. We howled, drank and had quite a party till late night. And I am at my home now writing.. 🙂

Thoughts with my espresso shot

Just waiting for my espresso shot at starbucks. I just woke up few minutes earlier and the first thing that I crave for when I wake up is sunlight and fresh air. So, I started biking in the campus; all the familiar roads – narrow and wide – it’s a beautiful sunny morning and I’m enjoying the refreshing outside. It’s probably soon going to be warm, so I’m happy not to miss this small window of comfort. The summer Purdue campus this year was a mess really, in fact, the whole West Lafayette was a mess with constructions with their screaching metallic sounds everywhere. Just to cross the Chauncey area street you would literally have to walk quarter miles to avoid sands and dirts. The nice bookshop there seems to be going away which is sad. The shop used to have lots of interesting stuff and I had some good memories with friends too. So, I’ll miss it. Last night was thirsty Thursday night which means I was hanging out with my friends at Cactus. 😉 I didn’t get too much wasted but sometimes those Thursday nights are just eventless and feel meaningless. But sometimes some night’s are unexpectedly awesome. But anyway, after a day of working in a quiet lab thinking about science and trying to understand the words in journals and papers and trying to understand the crazy scientists, the urge is just too strong to get lost in crowd and noise at night, at least for me. Anyway, coming back to the moment, I’m sitting right now with my espresso shot outside of the Starbucks at the nice flowery corner in front of the Krach Leadership center. I like this spot a lot. From here you can have a nice view of the recreational center and watching sunset from the other side in the evening is charming too. Even though I hate these constructions, but they were progressively making the campus look really newer and nicer. One thing that boggles everyone who is new around here and driving a car is those one way streets and some of them are now turned two way streets. Many many thanks.. There are some new food corners back there where I haven’t checked in yet even after three years of roaming around. See, there’s always things unnoticed. I’m almost done with my espresso shot now; in between writing my thoughts here, I already checked Facebook several times, already had a small online slack meeting with my student giving him instructions for his task and for sure, listening music. I am listening folk songs today, the whole last week I was listening rock and metals. It’s nice being engaged with things, with interactions with people; may be that’s one way of not getting bored. But then again you need to be with yourself too and see whether you are not being confused with these infinite stream of events around. It’s not nice feeling lonely too. I’ll have to admit that I do feel lonely even in the midst of plenty of people, even in the midst of handling so many things. It’s kind of interesting and may be it’s sort of a bad illusion of my mind that it often longs for the things which are missing but forgets to appreciate the things which are present. Living in the moment is not really easy after all. Mind always drags to the past or the future. And that’s why I like writing as it helps me think through.  I’ll have to move now, to some other corner of the campus doing something else. I love learning and I’ll engage myself to find some new ideas for today.

Fossil Record as a way to learn earth history

Who studies fossils? A Paleontologist. Why do paleontologists study fossils? Because fossil record brings information, provides clues, ideas about climatic changes of the planet, the evolution of geographical changes occurred on earth. For example, plant fossils and pollen fossils have been used to indicate climatic change of earth. To create geologic timescale, scientists have used fossils. We can see the fossil evolution during Paleogene period (over 66 Million years ago) in the image below. You can see the branches and subbranches in the image. The Subbotina trivialis (genus: Subbotina, species: trivialis) is highlighted in red.

Screen Shot 2017-07-26 at 9.19.52 PM
Evolution of Planktonic fossils during Paleogene period. The fossil image of Subbotina Trivialis has been shown in the rectangular yellow window. Subbotina Trivialis has found around 65.5 Million years ago.

Determining the age of fossil is very important and very challenging. Fossils are very often found in rocks and comparing one rock formation with another (relative dating), it’s possible to find a relative age for a fossil. Dating rocks involve calculating the rate of decay of radioactive element such as Carbon-14, Uranium-238, Potassium-40, Aluminium-26, Samarium-147, rubidiam-87, strontium-87. Fossilization is a rare event as there may not be any trace of an organism after its extinction. Therefore the record of an organism as the record of life in a fossil is something very significant to discover. The organism’s physical structure and subsequently deduced information such as it’s environment, diet, life-cycle can be obtained by studying fossils. Trace fossils, or fossilized marks left as a result of the activities of creatures such as trails, footprints, and burrows are also recorded and used as the source of information. From the fossil records throughout geologic time, scientists understood that the evolution of life is not a linear process. Sometimes the process is slow and sometimes it’s exponential. We also discovered that there might be periodicity in mass extinction by studying fossil records. Even the concept of plate tectonics was helped by fossil records. The more I am learning about fossils, the more exciting it’s becoming.

Eureka Moments

Every time I start reading a scientific paper, after reading the first page, my mind is already distracted and flying around everywhere trying to escape the jargons and heavy details proving my cognitive dissidence, making me doubt myself about whether I have autism or ADHD, then I need to like pull myself backward, reread, re-realize, then after reading another ten pages that way, I am like where did I start and where I stand now; I am in the dark; so I am lost, But wait.. then a moment like this occurs all on a sudden out of nowhere; it’s like a flashlight right on your face, everything is clear like a sunny day, every word starts making sense.. And I feel like running outside naked saying Eureka like Archimides when I can not stop feeling awed by the marvelous results and the implications and the beauty of science that strike me like a lightning.

The Red Converse

He is walking on the street. The warm afternoon is somewhat helping him think clearly. The stores around here are quite familiar. The bookstore in the corner still looks the same. There was a shoe store right opposite to the office building near the crossing but it is now replaced by a pizza shop. Now he suddenly becomes sad. He just remembered the day when he bought a converse for her from that store. They both were walking on this street and they stopped right in front of the ice cream stall in front of the building when they could see the store across the street. He can still visualize that afternoon but may be it would have been easier to remember some details if the store was still there. He closes his eyes. She chose red color and it was a nice looking converse. He couldn’t remember whether she was already decided to buy red before they came to the store or was it that specific red converse that took all her attention? She probably doesn’t have that red converse anymore. He tried to think the last time she wore it. But he couldn’t remember. All he can think is her smiley face when she was trying it. She probably also jumped a bit to see how comfortable it would feel while running. The guy in the store was bringing more converse but he might have also been convinced that she would go for the red one from her expression. She asked him how she looked on the red converse. He just smiled and she knew what it meant. She also smiled back. He now turns around because he didn’t want to remember more. He walks straight towards the smokeshop. It’s been a while he wasn’t smoking but now he feels the need for a distraction. Nicotine helps.

Power?

What is that define power? Is it the ability of doing something? Is it the acknowledgement? How does the transformation from the powerlessness to powerfulness occur? Is it that power already lies inside us? Is it that we are always potentially powerful? Only when the event of the demonstration of power happens, we realize it? But people seem to actively seek for power as if it lie’s somewhere outside. Is money power? It seems everyone believes having more money will give them more power. Is the purchasing power equivalent to the psychological power? Given that we live in a knowledge, information economy, does the acquisition of more information make us more powerful? What is the main semantic difference between power and energy? I am energetic vs I am powerful- they sound different, why? What gives you energy vs what gives you power? Is energy more like a physical thing, whereas power is more psychological? Do you choose to be powerful? Do you choose to be powerless? Is it a zero sum game?

Music, Hippy & Spirituality

I find music as something that can give some bliss, satisfy me. It definitely depends on what you listen and how you let music touch your heart. But there’s always a kind of music that can fit with your mood. There are so many amazing talents in the world of music who want to reach through their vocal genius. I am always drawn to a song with good lyrics and good music. Even though I have fascination for rock music, I try to listen every genre as long as it satisfies the two criteria- some good poetic, musical, rhythmic words and the awesome enchanting music – which means I feel the joy. If I think with which I have spent most of my young and adult life- during my time of joy and sorrow, it will definitely be some form of music. Some music just refresh me when I wake up, some just push me to work a little bit harder when I almost give up, some are inevitable in my tired evenings and some just can soothe me and put me to sleep. When I was at my late teen years, when I started exploring the world, when I left my country and started my traveller life, it was the music which accompanied me on my way, it was those headphones that I used to wear everywhere and shake my head in a crowdy train platform surprising everyone. I was always a boy who loves to express where ever he feels to. I felt like a hippy so many times, I discovered the bohemian in me through the music, I realized my spiritual connection with nature through music. Some music are imprinted in my brain so deeply, I can just close my eyes and I can time travel from one corner of the world to other corner, I can fully explore my imaginary world, music works for me as an emotional vehicle, gives me wings to fly, fly higher. I am lucky to have learnt several languages like Bengali, English, Japanese, Hindi- and all these four cultures admire music so much. I wish if I could add Spanish to this, that would be perfect. Anyway, I thoroughly enjoy the similarities and the differences of these different music of different culture. Bengali, being my mother tongue, touches me and caresses me. Most of my crucial teen years, when I was dashing, Japanese music gave me the strength, gave me the power to think broadly, inspired me to explore. And what can I talk about English music – they are just my life in one sentence. From classic rock to alternative rock to pop rock to country, everywhere I swing depending on my mood. And my young life back in Bangladesh was mostly surrounded with Hindi music. No wonder why my romantic me has so many different moods, lol.. I feel like a Bengal tiger, then a Japanese samurai, then an English traveller. I am going to share some music which are very integral to me, has always been with me for some very real reasons, remind of specific time and I will continue listening to them till my last breath.

1.

2.

3.

4.

5.

6.

7.

8.

9.

10.

11.

12.

13.

14.

15.

16.

Adversity can be blessing

Adversity is not always bad. In fact, adversity or lack of something provoke you to come up with some useful solution. If everything you need is right at your reach, it may feel comfortable but I am telling you that it’s delusional to think that state as a sustainable one. It’s not a stable situation where you don’t feel the lag or emptiness. It’s probably only an utopian state where you get everything ready, prepared, arranged or organized around you. So, the lack of something, the emptiness may bother us, frustrate us, bring anxiety to our psyche but then those same feelings force us to do something, do something to fill the gap or the void. And sometimes we may turn out to be really creative that can impress our own selves. Innovation at its heart depends on the desire to solve an issue, a problem and being passionate or motivated towards implementing those solutions. And when we have absence of something we truly want; the shortage, the scarcity of the elements, materials or instruments; the unavailability or deficit of the required things turn on our discoverer us to flow, move, interact, look and experiment. It’s not always alright to have vertical structure, sometimes we need to inspect horizontally. Going deeper and deeper may sometimes make us at a loss when we need to look widely. One important thing adversity tells us that you can solve anything you want with only the things that you have at hand. Focus on the things that you have and carefully examine, investigate whether there’s a possible scenario to create. When you are smart, adversity is blessing in disguise. I realize more now the words of Steve Jobs

Stay hungry. Stay foolish.