I can't believe I made it this far !!!

iteration 1
nodejs/mongo db

abysmal failure

but that's ok i guess
i did my initial assessment / proof of concept using a new platform (and landed a dev gig - pure dumb luck there)

i did pick up a little nodejs/pure javascript stuff so it wasnt a total loss

so it was only half a failure i guess

iteration 2
port everything to php / mysql

got it rocking fairly fast

my old multi-curl scraper was broken - i had fixed it / recoded it once before
but some asshole stole my dev computer that had good code on it

so i lost like 3 or 4 days troubleshooting and fixing it again

that turned out pretty good - because now it actually works - and much better than my older fixes

iteration 3

trying to fix up the loose ends

making a half decent UI is kicking my butt - i am definitely not a front-end guy LOL

problems
    using older bootstrap version with an even older bootstrap css 
    roflmao - no wonder everything i tried looked horrible
    
    i had to rip through all my old crap code and get everything up to date for 
    jquery / bootstrap
    
    wow - amazing - everything actually works like it should for once
    
    next problem - 
    i forgot to update my MAX_FILE_SIZE upload in php.ini settings      
    and that still wasnt working - 
    duh - forgot to restart my apache service so the setting weren't active
    so all my uploads still weren't working
    
    yay - problem solved after apache reboot - im such an idiot

here we are in the home stretch

i have the command line interface running awesome fast

now that i have the right UI / jquery / bootstrap / uploader stuff working i should be able to
just plug everything in to the UI fairly quick from this point.

i cant believe some of the stupid problems i have been having with this

on a positive note - all that broken stuff that has been slowing me down is fixed fixed fixed ;)

upcoming @todo list

multi engine
currently the scraper just does one site at a time - in succession

i can get this to run a little better if i spread scraper across all domains

i did it once before - so shouldn't be too hard - just need my brainz to cooperate

ajax api call
damn NDA - can't talk about this part
finish the UI - get stats and charts working again
what good is all this data if nobody can get to it or use it

I probably don't have to do all the 'ajaxy' stuff -
but what the hell , it just adds to the fun of the challenge...

part of this is adding in some more documentation (the database design)

once again - I am going to shoot for monday launch - this should be interesting

crap - lost another 3 hours fighting the damn dev server
apparently it was still running IIS
so i had to jumble turning it off
then the friggin apache didnt want to load php - so i had to battle that crap
oh look - let's fix the firewall crap while were at it LOLZ

Well, i hope i can get this thing up and running in the next week...