You and I

I: I like that you like to live in the moment.

You: We are always living in a moment. What do you mean? You think you are not living in the moment?

I: True. But what I mean is that if I am doing things now for something that I expect to happen in a future moment, am I living in the moment? I am bit confused.

You: Of course you are still living in the moment. But you probably don’t feel like living in the moment.

I: But if I am not feeling to be living in the moment, am I truly living in the moment? Isn’t there a distinction between fact and fiction? Or is it that everything is fiction?

You: It can be that everything is a fiction.

I: No, our reality, the world outside us might not be a fiction. Pain might not be a fiction. Death might not be a fiction. I am not sure about pleasure though. Pleasures can be fictitious. I don’t know.. You seem to be living in the moment whereas I can’t really claim it like that. Or I should say I don’t feel to claim it like that. I tend to feel that I know what is ahead of me. At least I can have a good prediction of that based on my prior experience. But I am relying on my thinking mind and my collected knowledge and my instinct.

You: But don’t you ever wonder to start your day like you are a newborn baby?

I: I actually do.

You: Have you asked yourself why you have that desire to be reborn? Is it because you are burdened with your past?

I: It can be. But I’d say it’s not as simple as that.

You: I saw an episode of Black Mirror the other day. One thing kind of struck me. They were saying that memory is a way to trigger you to get back to your past to change your path. I have never thought it that way?

I: In a way it’s true. Especially if you are taking an action based on your memory. Isn’t it?

(to be continued ..)

Looking Inside Eminem’s Lyrics – part 1

I started analyzing the lyrics of Eminem. My initial interest is that what are the most common words Eminem has been using in his lyrics. I collected the name of all his 287 songs from this link. Then I collected the lyrics using python sontext library which collects lyrics from I have become successful in gathering lyrics for 223 songs of Eminem out of those 287 song names using my python code. After gathering the lyrics, I had to process the text in the lyrics. Normally in language processing tasks, we do chunking, lemmatization, stemming, spelling error check. I have used NLTK library for all these. Actually I had to avoid doing lemmatization as it was chopping off lots of interesting words in its existing form. And also I created a banned words list as Eminem has used a lot of ‘na’, ‘wa’, ‘oh’ kind of words which semantically doesn’t have much meaning.  Then I used NLTK word frequency method to find out the frequency of words. The top 20 words used were

[(u’like’, 1375), (u’get’, 1049), (u’got’, 907), (u’shit’, 740), (u”’cause”, 729),

(u’know’, 701), (u’back’, 674), (u’fuck’, 671), (u’eminem’, 593), (u’go’, 557),

(u’see’, 514), (u’one’, 497), (u’say’, 476), (u’never’, 430), (u’bitch’, 428),

(u’man’, 428), (u’let’, 422), (u’time’, 411), (u’come’, 392), (u’think’, 361)]

And yeah apparently Eminem has cursed a lot in his songs. As you can see in the plot below for the word “shit” (rank:4), “fuck” (rank:8), “bitch” (rank:15).

Frequency of top 20 words

Then the word ‘love’ has been used 282 times just bit less than the word ‘ass’ which was used 295 times. You can see the word ‘dre’ has been used a lot and it’s most likely Dr. Dre who worked with Eminem. The word ‘man’ is used more than the word ‘girl’. The word ‘hate’ is used less than the word ‘love’, only 116 times. Here’s two more plots for word frequency.

Frequency of top 50 words
Frequency of top 100 words

Anyway, simple bag of words probably don’t give good representation of a particular song. For example, the word love can be used in a sentence “I love you” but then also “I don’t love you” which has completely opposite meaning. But here they are counted all together. Before contextual analysis, I was just thinking about doing another frequency analysis according to Russel’s model of mood. You basically divide the xy-plane into four orthogonal regions as you can see in the image below.

Model of mood

I want to see where eminem’s music in general fall in this emotional plane. There’s more interesting analysis I can do later on using word vector and other new NLP techniques. I’ll eventually look into other artists, other genres and try to find whether there are different patterns in how the words are chosen and the kind of emotion certain songs may generate.

Code for getting Lyrics:

import lxml
from lxml import html
import requests

import pickle

import numpy as np
import libsongtext
from libsongtext import lyricwiki
from libsongtext import utils

import pprint as pp

artist_name = 'eminem'

url = ''
#f = urllib.urlopen(url)
f = requests.get(url)

html_page =
tree = html.fromstring(html_page)

song_name_xpath=tree.xpath('//div[@class="article-content clearfix"]//strong/a')

num = 1
lyrics_list = {}
lyrics_not_found_list = []
success_lyrics_cnt = 0
for s in song_name_xpath:
song = ''.join(s.text.encode('ascii', 'ignore').strip())

print 'No. ' + str(num)
num = num + 1
print 'track : ' + song

args = {}
args['artist'] = artist_name
args['title'] = song.strip()
title = args['title']
if not lyrics_list.get(title):
t = lyricwiki.LyricWikiSong(args)
lyrics = t.get_lyrics()
print "Got Lyrics."
lyrics_list[title] = lyrics
success_lyrics_cnt += 1
print "Failed to get Lyrics."

print 'Successfully got ', success_lyrics_cnt, ' lyrics out of ', len(song_name_xpath), ' tracks'

def save_obj(obj, path, name):
with open(path + name + '.pkl', 'wb') as f:
pickle.dump(obj, f, pickle.HIGHEST_PROTOCOL)

def load_obj(path, name):
with open(path + name + '.pkl', 'rb') as f:
return picle.load(f)

save_obj(lyrics_list, '/Users/andy/Documents/projects/music/lyrics_database/eminem/', 'eminem_song_lyrics')

Code for word frequency analysis in Lyrics:

import pickle
import string

import nltk
from nltk.tokenize import sent_tokenize
from nltk import word_tokenize
from nltk import sent_tokenize
from nltk.corpus import stopwords

import enchant
from enchant.checker import SpellChecker

eng_dict = enchant.Dict("en_US")

#import lyrics of eminem
f = open(eminem_lyrics_pickle_file, 'rb')

lyrics= lyrics_list.values()

# english words
#words = set(nltk.corpus.words.words())

porter = nltk.PorterStemmer()
wnl = nltk.WordNetLemmatizer()

def plot_freqdist_freq(fd,
title='Frequency plot',
As of NLTK version 3.2.1, FreqDist.plot() plots the counts
and has no kwarg for normalising to frequency.
Work this around here.

- the FreqDist object
- max_num: if specified, only plot up to this number of items
(they are already sorted descending by the FreqDist)
- cumulative: bool (defaults to False)
- title: the title to give the plot
- linewidth: the width of line to use (defaults to 2)
OUTPUT: plot the freq and return None.

tmp = fd.copy()
norm = fd.N()
for key in tmp.keys():
tmp[key] = float(fd[key]) / norm

if max_num:
tmp.plot(max_num, cumulative=cumulative,
title=title, linewidth=linewidth)


stem_tokens = ['ed', 'ies', '\'s' , 'n\'t', '\'m', '--', '\'\'']
banned_words = ['ha', 'wa', 'ta', 'u', 'i', 'ai', 'na', 'ca', '...', '..', '\'em', '\'en', 'wan', '`', '``',
'oh', 're', '\'re', '\'ne', 'yea', 'yeah', 'ya', 'yah', '\'ve', '\'d', 'wo', 'oh', 'ooh',
'\'ll', 'yo', 'is\\u2026', 'ah', 'wit', 'would', '\\u2019']

#['i\'ma', 'y\'ll']

def synonyms(word):
syns = []
for word in wn.synsets(word):
sim_words = word.similar_tos()
sim_words += word.lemma_names()
for sim in sim_words:
s = sim
if hasattr(s, '_name') :
s = sim._name.split(".")[0]

syns = set(syns)
return syns

def stem(word):
for suffix in stem_tokens:
if word in banned_words:
return False

if word == 'suffix' or word.endswith(suffix):
return word[:-len(suffix)]
return word

lyrics_edited = []
chkr = SpellChecker("en_US")

edited_tokens = []
i = 1
for s, l in lyrics_list.items():
print i, ". Processing song: \"", s, "\""
i += 1
# find wrongly spelled words
for err in chkr:

tokens = word_tokenize(l)
l_txt = nltk.Text(tokens)

for t in tokens:
tn = t.lower()
#tn = porter.stem(t)
#tn = wnl.lemmatize(tn)

tn = stem(tn)
if tn and tn not in err_words and tn not in stopwords.words('english') and tn not in list(string.punctuation):

uniq_tokens = set(edited_tokens)

fdist = nltk.FreqDist(edited_tokens)

#Rusell's Model of mood
mood_happy_words = ['Exhilarated', 'Excited', 'Happy', 'Pleasure']
mood_h = []
for ws in mood_happy_words:
for w in synonyms(ws):

mood_h = list(set(mood_h))

mood_angry_words = ['Anxious', 'Angry', 'Terrified', 'Disgusted']
mood_a = []
for ws in mood_angry_words:
for w in synonyms(ws):

mood_a = list(set(mood_a))

mood_sad_words = ['Sad', 'Despairing', 'Depressed', 'Bored']
mood_s = []
for ws in mood_sad_words:
for w in synonyms(ws):

mood_s = list(set(mood_a))

mood_relaxed_words = ['Relaxed', 'Serene', 'Tranquil', 'Calm']
mood_r = []
for ws in mood_relaxed_words:
for w in synonyms(ws):

mood_r = list(set(mood_r))

“The mind is its own place, and in itself / Can make a Heaven of Hell, a Hell of Heaven.”

I am seriously thinking about starting meditation again. The peace of mind is quite rare to obtain these days. I find myself very easily distracted, find it really hard to concentrate, find it hard to remember, find it hard to forget. I was reading about meditation and it’s different form. I loved some of Sam Harris’s quote (original post) and am reposting. I will elaborate the part that intrigued me when I have some time later on.

Although the insights we can have in meditation tell us nothing about the origins of the universe, they do confirm some well-established truths about the human mind: Our conventional sense of self is an illusion; positive emotions, such as compassion and patience, are teachable skills; and the way we think directly influences our experience of the world.


The conventional sense of self is an illusion [and] spirituality largely consists in realizing this, moment to moment. There are logical and scientific reasons to accept this claim, but recognizing it to be true is not a matter of understanding these reasons. Like many illusions, the sense of self disappears when closely examined, and this is done through the practice of meditation.


The feeling that we call “I” seems to define our point of view in every moment, and it also provides an anchor for popular beliefs about souls and freedom of will. And yet this feeling, however imperturbable it may appear at present, can be altered, interrupted, or entirely abolished.


The self that does not survive scrutiny is the subject of experience in each present moment — the feeling of being a thinker of thoughts inside one’s head, the sense of being an owner or inhabitant of a physical body, which this false self seems to appropriate as a kind of vehicle. Even if you don’t believe such a homunculus exists — perhaps because you believe, on the basis of science, that you are identical to your body and brain rather than a ghostly resident therein — you almost certainly feel like an internal self in almost every waking moment. And yet, however one looks for it, this self is nowhere to be found. It cannot be seen amid the particulars of experience, and it cannot be seen when experience itself is viewed as a totality. However, its absence can be found — and when it is, the feeling of being a self disappears.


We wouldn’t attempt to meditate, or engage in any other contemplative practice, if we didn’t feel that something about our experience needed to be improved. But here lies one of the central paradoxes of spiritual life, because this very feeling of dissatisfaction causes us to overlook the intrinsic freedom of consciousness in the present. As we have seen, there are good reasons to believe that adopting a practice like meditation can lead to positive changes in one’s life. But the deepest goal of spirituality is freedom from the illusion of the self — and to seek such freedom, as though it were a future state to be attained through effort, is to reinforce the chains of one’s apparent bondage in each moment.


The ultimate wisdom of enlightenment, whatever it is, cannot be a matter of having fleeting experiences. The goal of meditation is to uncover a form of well-being that is inherent to the nature of our minds. It must, therefore, be available in the context of ordinary sights, sounds, sensations, and even thoughts. Peak experiences are fine, but real freedom must be coincident with normal waking life.


Those who begin to practice in the spirit of gradualism often assume that the goal of self-transcendence is far away, and they may spend years overlooking the very freedom that they yearn to realize.


[This approach] encourages confusion at the outset regarding the nature of the problem one is trying to solve. It is true, however, that striving toward the distant goal of enlightenment (as well as the nearer goal of cessation) can lead one to practice with an intensity that might otherwise be difficult to achieve. I never made more effort than I did when practicing under U Pandita. But most of this effort arose from the very illusion of bondage to the self that I was seeking to overcome. The model of this practice is that one must climb the mountain so that freedom can be found at the top. But the self is already an illusion, and that truth can be glimpsed directly, at the mountain’s base or anywhere else along the path. One can then return to this insight, again and again, as one’s sole method of meditation — thereby arriving at the goal in each moment of actual practice.


It is very difficult to imagine someone’s not being able to see her reflection in a window even after years of looking — but that is what happens when a person begins most forms of spiritual practice. Most techniques of meditation are, in essence, elaborate ways for looking through the window in the hope that if one only sees the world in greater detail, an image of one’s true face will eventually appear. Imagine a teaching like this: If you just focus on the trees swaying outside the window without distraction, you will see your true face. Undoubtedly, such an instruction would be an obstacle to seeing what could otherwise be seen directly. Almost everything that has been said or written about spiritual practice, even most of the teachings one finds in Buddhism, directs a person’s gaze to the world beyond the glass, thereby confusing matters from the very beginning.

But one must start somewhere. And the truth is that most people are simply too distracted by their thoughts to have the selflessness of consciousness pointed out directly. And even if they are ready to glimpse it, they are unlikely to understand its significance.


Embracing the contents of consciousness in any moment is a very powerful way of training yourself to respond differently to adversity. However, it is important to distinguish between accepting unpleasant sensations and emotions as a strategy — while covertly hoping that they will go away — and truly accepting them as transitory appearances in consciousness. Only the latter gesture opens the door to wisdom and lasting change. The paradox is that we can become wiser and more compassionate and live more fulfilling lives by refusing to be who we have tended to be in the past. But we must also relax, accepting things as they are in the present, as we strive to change ourselves.


Happiness and suffering, however extreme, are mental events. The mind depends upon the body, and the body upon the world, but everything good or bad that happens in your life must appear in consciousness to matter. This fact offers ample opportunity to make the best of bad situations — changing your perception of the world is often as good as changing the world — but it also allows a person to be miserable even when all the material and social conditions for happiness have been met. During the normal course of events, your mind will determine the quality of your life.


The human nervous system is plastic in a very important way — which means your experience of the world can be radically transformed. You are tending who you were yesterday by virtue of various habit patterns and physiological homeostasis and other things that are keeping you very recognizable to yourself, but it’s possible to have a very different experience… It’s possible to do it through a deliberate form of training, like meditation, and I think it’s crucial to do — because we all want to be as happy and as fulfilled and as free of pointless suffering as can possibly be. And all of our suffering, and all of our unhappiness, is a product of how our minds are in every moment. So if there’s a way to use the mind itself to improve one’s capacity for moment-to-moment wellbeing — which I’m convinced there is — then this should be potentially of interest to everybody.