Back to Posts
Listen to Thread

Is it posible to fetch minute data ?

https://www.quantopian.com/algorithms/5162617f7eee96386f00002e

Somehow date_format='%Y-%m-%d %H:%M:%S' is not working. Is only date formatting supported ?

Hi Avinash, the full format should be supported. Could you paste a few lines of the file here? If you wrap the post in triple single quotes it will maintain indentation:

        like this  

thanks,
fawce

This is sample file

Date_Time,open,high,low,close  
2013-02-01 00:00:00,1.3612500000000001,1.3612500000000001,1.3607100000000001,1.3607799999999999  
2013-02-01 00:01:00,1.3607899999999999,1.3608499999999999,1.36066,1.36077  
2013-02-01 00:02:00,1.36077,1.3608,1.36076,1.36076  
2013-02-01 00:03:00,1.36076,1.3608100000000001,1.3606799999999999,1.3607100000000001  

Code Snippet

import pandas

def rename_col(df):  
    df = df.rename(columns={'close': 'price'})  
    df = df.fillna(method='ffill')  
    df = df[['price', 'sid']]  
    log.info(' \n %s ' , df.head())  
    return df  
def initialize(context):  
    fetch_csv('https://raw.github.com/avinashpandit/forexdata/master/eurusdx00.csv',  
        date_column='Date_Time',  
        symbol='eur/usd',  
        usecols=['close'],  
        post_func=rename_col,  
        date_format='%Y-%m-%d %H:%M:%S'  
        )  
    context.stock = sid(17111)

def handle_data(context, data):  
    if 'price' in data['eur/usd']:  
        print(data['eur/usd'].price)  

Hi Avinash,

This is a pretty interesting problem. The security you are using, sid(17111), only has trading history in 2002, so the backtester will automatically suggest that you shift your test period to 2002. However, your sample file has data for just a few days in 2013. So the automatic shift moves the test outside the range of your file. If you move the test period to 2/1/2013 - 2/4/2013, you should see your print statement in the logs.

thanks,
fawce

Thanks John. My bad i always used to select period suggested instead of my own period.

Hello All,

I'm trying to replicate this with different data:

Time,Open,High,Low,Close,Volume  
01.11.2012 00:00:00,1.29642,1.2965,1.29642,1.29643,18.67  
01.11.2012 00:01:00,1.29643,1.29647,1.29641,1.29643,36.43  
01.11.2012 00:02:00,1.29647,1.29651,1.29634,1.29641,49.61  

and this code:

import pandas

def rename_col(df):  
    df = df.rename(columns={'Close': 'price'})  
    df = df.fillna(method='ffill')  
    # df = df[['price', 'sid']]  
    log.info(' \n %s ' , df.head())  
    return df  


def initialize(context):  
    fetch_csv('http://clacton1.no-ip.biz/EURUSD 1 Minute 1.11.12 To 17.11.12.csv',  
        date_column='Time',  
        symbol='eur/usd',  
        usecols=['Close'],  
        post_func=rename_col,  
        date_format='%d.%m.%Y %H:%M:%S'  
        )  
    context.stock=sid(24)

def handle_data(context, data):  
    if 'price' in data['eur/usd']:  
        print(data['eur/usd'].price)  

but I get an error relating to the date formatting, I think:

ImportError: cannot import name universaldetector  
File test_algorithm_sycheck.py:17, in initialize  
File algoproxy.py:960, in fetch_csv  
File requests_csv.py:210, in __init__  
File requests_csv.py:228, in load_df  
File requests_csv.py:224, in fetch_data  
File models.py:608, in text  
File models.py:513, in apparent_encoding  
File __init__.py:27, in detect  

The data sample ha 24,481 rows.

Regards,

Peter

Hi Peter,

Thanks for reporting this issue. The problem seems to be that we don't have the library chardet.universaldetector available for your algorithm, and it therefore isn't able to check the encoding on your csv file. We'll get it loaded into the environment soon.

In the meantime, I just tried another csv file without a problem. Maybe there is a way to convert your file to avoid the call to universaldetector?

thanks,
fawce

Hello John,

Thanks for the prompt response. I've wasted hours on this trivial test....it seems it's not the file. I was running the Python HTTP server on my home machine:

python -m SimpleHTTPServer

which seemed to be working. But I now find the same file when served from github works without an issue i.e.

fetch_csv('https://raw.github.com/pcawthron/ForexData/master/EURUSD%201%20Minute%201.11.12%20To%2017.11.12.csv',  

works, but

fetch_csv('http://clacton1.no-ip.biz/EURUSD%201%20Minute%201.11.12%20To%2017.11.12.csv',  

fails with an apparent encoding error. Apologies for wasting your time.

Regards,

Peter

Peter,

It is not a problem - we really should have that library installed. Maybe your local server is not setting the mime/type for .csv extension?
Glad you found a work around - I'll let you know when we get universaldetector loaded, so you can try your local setup again.

thanks,
fawce

Hi Peter,

I saw you have fx data with volumes. From where you got fx data with volume  ?

Thanks,
Avinash

Hello Avinash,

It's from Dukascopy who also have tick data:

Time    Ask Bid AskVolume   BidVolume  
01.04.2013 01:00:01.515 1.27926 1.27919 1.35    4.35  
01.04.2013 01:00:01.965 1.27926 1.27921 1.13    1.5  
01.04.2013 01:00:02.266 1.27929 1.27924 1.5 1.73  
01.04.2013 01:00:02.325 1.27927 1.27924 1.35    1.5

See: http://www.dukascopy.com/swiss/english/marketwatch/historical/

Regards,

Peter

Thanks Peter.

I reproduced the problem by setting up a CGI script to serve a CSV file with type application/octet-stream rather than text/csv, so that is indeed the issue. I've opened an issue about this for us to track it internally. In the meantime, if you just make sure whatever is serving your CSV file serves it with text/csv, you should not encounter this issue.

Hello Jonathan,

My hours weren't wasted then. I'll call it testing!

Regards,

Peter

Log in to reply to this thread.
Not a member? Sign up!