Back to Community
Market Sentiment, Market mood, FinSentS signals detection

Hi all,

I am creating this post as I am interested in a particular hot topic:
- Market mood
- Market sentiment
This subject can be easily leveraged via Algo Trading. Hedges funds are already doing that.

The real question is “How easy is it?”

From a previous post Google Search Terms predict market movements; we saw few positives results on Google search words and Wikipedia search trends hold signals that can predict the markets moves.

When I was reading the post I detected that major challenges were:
- How to extract Google trends and Wikipedia figures?
- Where to find the data itself?
- And how to extract it to make it available to our engine?

That is the oldest problem that is being faced per statisticians, econometrics people.
This problem is part of the “Big Data” issue. One of the other parts is the storage of this massive data.

FinSentS' statistical and semantic engine scans and indexes a thousand of financial sources to capture the mood of the market can be an alternative or solution to this problem.

Let’s see if there are signals from them. They have provided their historical data or few stocks or index.
You can test the data from the web application

http://portal.finsents.com/

Daniel

23 responses

Just for testing. ...

I cloned the algo "Google Search Terms predict market movements",
but instead of using Google Search, I am using the 1 month of News sentiment data on Apple from historical FinSentS data.

import numpy as np  
import datetime

Average over 2 weeks, free parameter.

delta_t = 5  
def initialize(context):

This is the search query we are using, this is tied to the csv file.

context.query = 'Sentiment'

Use fetcher to get data.
News sentiment data extracted from the files and format it in another excel file. This is purely a test data, please feel free to use.

Note: To my understanding it is a daily score that represent average of sentiment scores of Articles mentioning Apple in Business and Financial News.

fetch_csv('http://data.infotrie.com/Quantopian_sentiment_apple5_1daylag.csv',  
           date_format='%Y-%m-%d',  
          date_column='Date',  
          headers={'User-Agent': 'Blah'},  
          symbol='Sentiment',  
)  
context.order_size = 1000  
 context.sec_id = 24  
context.security = sid(24) # Apple  

def handle_data(context, data):  
c = context  

if c.query not in data[c.query]:  
    return  

Extract Sentiment scores of search query.

indicator = data[c.query][c.query]  

Buy and hold strategy that enters on the first day of the week and exits after one week.

#   if data[c.security].dt.weekday() == 0: # Monday  
# Compute average over weeks in range [t-delta_t-1, t[  

mean_indicator = mean_past_queries(data, c.query)  
if mean_indicator is None:  

#log.info('error1')  

return

Exit positions

amount = c.portfolio['positions'][c.sec_id].amount  
log.info('amount before')  
log.info(amount)  
log.info(mean_indicator)  
log.info(indicator)  

if (amount < 0) and (mean_indicator < indicator):  
    log.info('cover short')  
    order(c.security, -amount)

if (amount > 0) and (mean_indicator > indicator):  
    log.info('cover long')  
    order(c.security, -amount)  

# amountA = c.portfolio['positions'][c.sec_id].amount  
# log.info('amount after')  
# log.info(amountA)

Long or short depending on whether sentiment is clearly down or up, respectively.

#if indicator > mean_indicator and indicator < 35:  
if indicator < 35:  

#if (indicator < mean_indicator) and (indicator < 35):  
   order(c.security, -c.order_size)  

else:  
   if indicator > 70:

# if (indicator > mean_indicator) and (indicator > 70):  
         order(c.security, c.order_size)  

@batch_transform(window_length=delta_t+1, refresh_period=0)
def mean_past_queries(data, query):
# Compute mean over all events except most current one.
# log.info('error2')
return data[query][query][:-1].mean()

This is quite simple and not very realistic...

That sounds like an interesting data set and algo. Did you get it to run? I'd love to see the backtest.

Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

Hi Dan

Back test in there

Clone Algorithm
39
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 51dcc5ad48b09e06d3262ed4
This backtest was created using an older version of the backtester. Please re-run this backtest to see results using the latest backtester. Learn more about the recent changes.
There was a runtime error.

Interesting stuff. I definitely look forward to seeing more.

I modified your test a little. I shortened the test period (just so the graph would be better) and I put in some rudimentary borrowing limits.

Clone Algorithm
27
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 51dddcef71c34e06c3d71be1
This backtest was created using an older version of the backtester. Please re-run this backtest to see results using the latest backtester. Learn more about the recent changes.
There was a runtime error.

@Dan, in your backtest, see how context.portfolio.cash increases when you short the stock. How does that make sense? That's the kind of thing I was talking about in the other thread.

Hi Dennis,

When you short a stock, your cash balance does indeed increase, because you get the cash proceeds of the short sale.

The cash attribute on the portfolio doesn't represent the value of your portfolio, it represents cash on hand. For total value, you need to look at the pnl attribute on the portfolio.

See the attached backtest, which demonstrates this by buying one share of Apple per day during a month when Apple went down, and graphing various attributes of the portfolio object throughout the month.

jik

Clone Algorithm
18
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 51deda3b07e9ee06ce054f2f
This backtest was created using an older version of the backtester. Please re-run this backtest to see results using the latest backtester. Learn more about the recent changes.
There was a runtime error.
Disclaimer

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by Quantopian. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. No information contained herein should be regarded as a suggestion to engage in or refrain from any investment-related course of action as none of Quantopian nor any of its affiliates is undertaking to provide investment advice, act as an adviser to any plan or entity subject to the Employee Retirement Income Security Act of 1974, as amended, individual retirement account or individual retirement annuity, or give advice in a fiduciary capacity with respect to the materials presented herein. If you are an individual retirement or other investor, contact your financial advisor or other fiduciary unrelated to Quantopian about whether any given investment idea, strategy, product or service described herein may be appropriate for your circumstances. All investments involve risk, including loss of principal. Quantopian makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances.

@JIK, I just re-read the Help & API page for portfolio.cash. You are correct, it defines portfolio.cash as "The current amount of cash in your portfolio."

Maybe I'm being naive but how does that definition help in the creation of a trading algorithm? A far more useful definition would be "The current amount of cash available in your portfolio."

Take a look at the attached backtest. I modified your "short Apple" backtest. My version is "short SPY, long SH". It shorts SPY and goes long SH every day. Notice how the built-in portfolio object provides no useful information.

By the end of the backtest here are the overall position values:

SPY: ($5,316.80)  
SH:   $5,368.00  

But the portfolio object has this to say:

capital_used: 190.43  
cash: 1190.43  
pnl: 241.63  
positions_value: 51.20  

Clearly I've used more than $10,000 in capital (not $190). And I've far exceeded the available cash which should have only let me commit to a $1000 position before needing to borrow on margin.

I have already found a way to calculate things using a different method in my algorithms. But your Help Docs and examples are promoting the use of portfolio.cash without explaining that it doesn't measure anything useful.

Dennis

Clone Algorithm
8
Loading...
Backtest from to with initial capital
Total Returns
--
Alpha
--
Beta
--
Sharpe
--
Sortino
--
Max Drawdown
--
Benchmark Returns
--
Volatility
--
Returns 1 Month 3 Month 6 Month 12 Month
Alpha 1 Month 3 Month 6 Month 12 Month
Beta 1 Month 3 Month 6 Month 12 Month
Sharpe 1 Month 3 Month 6 Month 12 Month
Sortino 1 Month 3 Month 6 Month 12 Month
Volatility 1 Month 3 Month 6 Month 12 Month
Max Drawdown 1 Month 3 Month 6 Month 12 Month
# Backtest ID: 51deddd0030fd506cb63cc32
This backtest was created using an older version of the backtester. Please re-run this backtest to see results using the latest backtester. Learn more about the recent changes.
There was a runtime error.

If folks are interested, we have real time financial sentiment data available to test. http://psychsignal.com/analysis

Our data correlates well against the VIX and two widely followed sentiment surveys put out by the NAAIM and AAII Have a look at the comparison http://psychsignal.com/survey

Feel free to get in touch with me [email protected]

Dennis - I definitely get the point that you're making, and I agree with it. I haven't yet put together the big picture of how we're going to implement the change. It's one of those things that I want to change as rarely as possible, and that means I have to work it through a number of future scenarios and make sure it works. I'm particularly focused on live trading and how to best represent to the algorithm what your broker will and won't permit. That's a long way of saying "yes, but it will take a bit."

James, that looks interesting, and I'll take a look.

Thanks Dan, we're very early... still building out the site and API but would love to have some early beta testers if anyone is curious. If not I'll come sneak back over in August when things are more polished!

Hi all,

I am following a particular stock on the Singapore Market SGX.
Singapore Telecom.

I have extracted the data for the last month and spent some time on them.

Please find from this link my analysis on excel :
https://dl.dropboxusercontent.com/u/99980873/Singtel%20Test.xlsx

I proceed step per step, below a brief description of my analysis

  • 1) Raw data

I have selected the interesting data per date

1) Sentiment score (End of day one)
2) Last spot price of the day
3) Number of Good words and bad words
4) Volume of transactions

  • 2)Data used for the graph

I have cleaned the data in order to produce a graphical view in order to detect strategies.

i.e:
In the second graph,
I kept only the sentiment score between [1;30] and [70;100]

To me, it could be the relevant signals.
Score = 0 means “No news, no articles”
Score between [40;70] are not significant "Sentiment is So So "

  • 3) and 4) Graph generation and tentative of strategies identification

I have produced a first graph and then a second one in order to identify signals

  • 5) Raw code and result on the strategies

With the identified signals, I have manually tested the strategy.

Result was not good !
For 1 month data and a simulation with 10.000 stocks, PNL (-1000 SGD)

  • 6) Adjustment on the strategies

I have challenged a bit the data and I found the below:

Sentiment score is being produced at midnight (German time), that is 6am Singapore time

I can then put my order at the opening time and put the reverse order at the closing time.

In Singapore, the market hours

Singapore Exchange (SGX)

Pre Open Session : 8.30am - 8.59am
Matching : 8.59am - 9.00am
SGX Trading Hours : 9.00am - 12.30pm
: 2.00pm - 5.00pm
Pre Close Session : 5.01pm - 5.05pm

Matching : 5.05pm - 5.06pm
No Orders Accepted on POEMS : 5.06pm - 5.15pm
Orders Accepted for next Trading Day : 5.15pm onwards

The adjusted strategy is now positive.

For 1 month data and a simulation with 10.000 stocks, PNL (+3600 SGD)
I will test it on 6 months data !

I will see how it can be coded in Quantopian and will update you

Many assumptions have been done for this analysis, for example:

  • Brokerage fees and clearing not considered in the models
  • Orders can be placed without issue few second after market opening and few second before Market closing
  • Market price is not extremely volatile at the opening and closing time

Cheers
Daniel

Hey Daniel,

I too was looking at the algorithm based on google search data. You have to excuse me, as I am very new at this and I was hoping you could answer a few basic questions about what and how you're amending the base platform of this original algorithm.

First, you used "fetcher" to acquire sentiment on apple. How and does one get this data say if I am curious using this same algorythm but with the price of Oil? I am lost. Please help.

Hello,

The financial sentiment score is being provided per a market data provider named FinSentS.

From their historical data,
I am not able to find any of the Oil underlyings ie :"Brent" "WTI"

As I am focus on Equities market. I can't help you on that.

You should try to contact them in order to request these data.

Daniel

Dear All,

Long time, I did not post ...
From below, you can find a new set of data on SP500 stocks

https://www.dropbox.com/s/4i7lttkoxnolnyl/New%20Set%20of%20Data%20for%20SP%20500.xlsx

Daniel

Dear All,

FinSentS Open Beta Web application has been released.
You can test the data from the web application

http://portal.finsents.com/

Daniel

Hi,

Because a lot of traders have requested a gold sentiment index.

As wishes,
https://www.dropbox.com/s/ghua8n52lq9fq5a/gold_sentiment.csv

G10 currencies will come asap.

Daniel

Dear All,

I have received the below update from INFOTRIE website.

D

Hello,

You probably have heard about the recent attribution of the 2013 Nobel Prize in Economics to three people: Eugene F. Fama, Robert J. Shiller and Lars Peter Hansen.
This is a true "Financial" Nobel Prize, and a praise for their research on how financial markets work and assets such as stocks are priced.
Allow me to make a parallel with what we are doing at InfoTrie FinSentS in the field of Sentiment Analysis, Big Data and Financial Engineering:

■ Firstly, let's take University of Chicago Professor Eugene Fama, known as the "father of the efficient markets hypothesis," which asserts that all information is efficiently priced into the markets making it incredibly difficult to profit off of trading in the short run. If Fama is fully right, if markets are really efficient, our Sentiment Analysis is worthless!

■ But if you listen to the other Nobel Prize, Yale Economist Robert Shiller, one of the "father of behavioural finance", then it is another story! Since is book "Irrational Exuberance" (2000) he has been well known for its ability to recognize asset bubbles. I don't know whether Shiller ever looked at Sentiment Analysis, but it for sure broadens the toolbox to analyse bubbles.

■ In a more neutral way, I wanted to thank Hansen for his development of econometric techniques for analyzing data and asset prices. He balances the position of the first two laureates, and that highlights the importance of the usage of statistical methods in today's Financial Engineering.

So regardless of whether you trust Fama, Shiller or simply use the techniques developed by Hansen,
I strongly invite you to make up your on mind on Sentiment Analysis, and register for free on our Open Beta Portal: portal.finsents.com

Thank you,
Best regards,
Frederic GEORJON
CEO of InfoTrie Financial Solutions,

Hi all, I wanted to share an interesting blog post BigML recently published using our data: http://blog.bigml.com/2014/05/09/predicting-stock-swings/

The summary is that our data is very useful in predicting volatility.. which is what our current client feedback has said all along.

I think in general it makes sense that any sentiment source (apart from our own) would be predictive of volatility rather than price. I'm not saying sentiment can't be price predictive but the low hanging fruit seems to be volatility. If the market is emotionally excited then volatility should follow.

If you want to play around with our data feel free to create an account here: https://psychsignal.com/users/sign_up then download a historical data set here: http://psychsignal.com/docs There's no cost or subscription involved.

James

FinSents is now on Quandl