'''
Extract sensors that has some readings. Missing ones are ok
but not missing everythings
'''


fdIn = open('subsetSensors.txt', 'r')
fdOut = open('subsetSensors_remMiss.txt', 'w')
fdOut2 = open('WStations.txt', 'w')

sIDs = {} #Some sensors are missing all the time
sTime = {} #Times that sensor readings are available
first = True
for line in fdIn:
    if first == True:
        fdOut.write(line)
        first = False
        continue
    l = line.split('\t')
    if l[0] not in sIDs:
        sIDs[l[0]] = [True] * 8
    if l[2] != '-1': sIDs[l[0]][0] = False
    if l[3] != '-1': sIDs[l[0]][1] = False
    if l[4] != '-1': sIDs[l[0]][2] = False
    if l[5] != '-1': sIDs[l[0]][3] = False
    if l[6] != '-1': sIDs[l[0]][4] = False
    if l[7] != '-1': sIDs[l[0]][5] = False
    if l[8] != '-1': sIDs[l[0]][6] = False
    if l[9] != '-1': sIDs[l[0]][7] = False

    t = int(l[1])
    if l[0] in sTime:
        td = t - sTime[l[0]][0]
        if td not in sTime[l[0]][1]: sTime[l[0]][1][td] = 1
        else: sTime[l[0]][1][td] += 1
        sTime[l[0]][0] = t
    else: sTime[l[0]] = [t, {}]
fdIn.close()

for i in sIDs:
    if True in sIDs[i]: 
        sIDs[i] = True #Mark to skip
        #print i
#Reread file
fdIn = open('subsetSensors.txt', 'r')
first = True
useSID = []
newSID = {} #New sensor IDs for remaining weather stations
maxInterval = -1
for line in fdIn:
    if first == True:
        first = False
        continue
    
    l = line.split('\t')
    if sIDs[l[0]] == True: continue #skip
    
    if l[0] not in newSID: newSID[l[0]] = len(newSID)
    sID = newSID[l[0]]
    if l[0] not in useSID:
        maxTime = -1
        maxCount = -1
        for j in sTime[l[0]][1]:
            if sTime[l[0]][1][j] > maxCount: 
                maxTime = j
                maxCount = sTime[l[0]][1][j]
        if maxTime == -1: 
            print 'Few samples at', l[0]
            continue
        if maxTime > maxInterval: maxInterval = maxTime
        fdOut2.write(str(sID) + '\t' + l[2] + '\t' + l[3] + '\t' + l[4] + '\t')
        fdOut2.write(str(maxTime) + '\n')
        useSID.append(l[0])
    tmpStr = str(sID)
    for i in range(1, len(l)): tmpStr += '\t' + l[i]
    fdOut.write(tmpStr)
print maxInterval
fdOut.close()
fdOut2.close()
