Twitter's SEO Oopsie
10 Apr2009
According to Google, twitter.com has 61 million pages indexed
Also according to Google, api.twitter.com has 20 million pages indexed. Pages which are identical to normal twitter.com/username pages.
Oops.
- In: Management|SEO
I was recently talking to the Yipit team on why I have no desire to take funding – and part of it was that it took away my ability to try out new stuff.
So to help out a friend, I’ve been playing around with the Twitter.com and its various API methods. When filing an IP whitelist request, what I thought would take days was pleasantly approved in 24 hours.
So while playing around with it, and their OAuth method, I came across a confusing bug – and after an hour of trying to figure out what the hell it was, I went to the forums. And it turned out to be a real nasty bug. A bug they deemed critical. And with that single click (the bug wasn’t just High Priority – it was Critical) I moved from the pantheon of the masses to the select few to make Twitter.com run around and try to figure out what went wrong.
But that really wasn’t the point. Twitter API support was fast. Blazing fast. Withing moments Doug and Matt were on it, and within 5 minutes of filing the bug it was claimed.
Twitter.com may be maddeningly slow at times, but I can see why so many apps around Twitter as springing up – those guys kick ass in making sure everything is being taken care of.
Heatmaps – Tough
1 Mar2009
I’ve been lately working on trying to generate heatmaps from an assortment of data. To say it has been tough is an understatement – I have tried Python, PHP, and Ruby libs/classes, and all have been too slow, too weak, and/or plain ugly (high peak is red, not blue nor fire-colored).
In the case of local, it is a combination of:
- Generating data in the format you need (x,y) coordinates with a related ‘value’ (the higher the value, the more obvious in the heatmap)
- Utilizing the data and being able to quickly and efficiently plot it onto an image
- The ability to create multiple levels and multiple images. For example, if using Google Maps, there are at least 5 levels that are important enough. Each tile image is 256×256, and each zoom increases the # of tiles by a factor of 4 (when you zoom in, what was one tile becomes 4). This is arguably the hardest part – eg if you only create by 256×256 grids, you miss location information just out of the grid. There might be a heavy heat-spot you might miss. And generating one big image is out of the question as you would exhaust memory. Using rough math, and exactly full zoom out, we have 12 256×256 grids. Another level of zoom, you end up with 48 images. Another zoom you end up with 192 images. Another zoom nets you 768, and one more gets you at 3072 images. One image that is 3072 tiles of 256×256 is 201,000,000 pixels. Even 768 tiles ends up being 50 million pixels. One image = no go.
- Conversion of (latitude,longitude) point into X,Y coordinates. In itself not difficult, but when you are trying to keep track of grids, and minimize CPU time spent, it becomes much more difficult.
Anyway – more than anything, this post was just on a challenge I’ve been working on. I haven’t solved it, but what I love is how much it brings local to life, and makes for another way of looking at data. For example, fast food joints in California:
Ideally I can get this damn fast enough that I can generate this in real-time and release as a lib. If not – oh well, a very interesting learning experience.
Matt Cutts was right …
23 Jan2009
The Omron Pedometer is pretty addictive. Once you start tracking how much you are walking (and especially graphing it), you really want to keep going.
Alas for me, worst time possible. After blowing out my knee (13 days after I am still limping), the pedometer arrived two days later – and it has pushed me too far. A few days ago I cracked 5000 steps. Next day, needing to break the previous high, I cracked 6000. The three days since then? Extreme soreness due to over-exertion, and barely breaking 1500 (which really is not all that much).
Anyway – this post is more about lauding the connection between ‘offline’ devices and how connecting it to the computer is making it much more useful. If I was Omron, I would actually build out an online hub – now that would be awesome. Create groups, show your progress to others, etc.
O, and they really need to support Mac – come on!
Update:
Well, just an update on the knee. Turns out my ACL is pretty much crushed, but the MRI will determine if it is completely torn (uhoh) vs 60-80% damaged. I’ve also greatly strained my inner-thigh ligament, and the meniscus in my knee. Over three weeks later and I’m still walking gimped. Humbug!
- In: Sweet 2.0
Speeding up Google's Crawl Rate
7 Jan2009
So Google now lets you control the max crawl rate through a nifty slider. While most sites start at a max-allowed rate of 0.5 pages/sec, I’ve been able to get a few sites to 10 pages/sec.
I had some large sites that had pretty low max rates (eg 1 page/sec), and it took me some figuring out how to get Google to go faster quicker – sitemap. By submitting a sitemap with a sizeable # of pages, we were able to boost from 1 page/sec to 2.5 pages/sec in roughly 96 hours.
Across all my sites Googlebot hits me at least a million times every day.
- In: Management|SEO|Sweet 2.0
in Toronto Jan 25 to Jan 30
28 Dec2008
I’ll be in Toronto for the full week of Jan 25 to Jan 30 – if you want to meet up, be sure to contact me.
- In: Management
I’ve come to an epiphany lately. It may seem obvious, but it truly changes the way you perceive things. The truth is that everything is local.
Think about it – what is synonymous with the internet? Sex. Travel. Weather. News. Sports. They are all
very local oriented (sex – craigslist and AdultFriendFinder … whose affiliate managers email me twice a day ugh).
Really – the point of this post is for you to try to see the local aspect of your day to day life. The internet is a woven tapestry of local connections – but too often we see the finalized product, not the individual (local) links that bring it all together.
I will come back to this.
Lifestream Plugin is now Public
15 Dec2008
We were testing it – and now it is out on iBegin.
Our own estimates are roughly 5000+ sites are using this plugin already. I expect it to go up a lot as we refine it and make it simpler to install.
- In: Management
Who am I?
15 Dec2008
The post title references a classic Jackie Chan movie, replete with awful (and hilarious) dialog combined with bizarre fight sequences involving props galore.
While most of what I talk about here (and really think about daily) revolves around the local space, I do a lot more than that. I’ve been building (proper) websites since roughly 2000, and did not enter the local space until 2005. Enthropia Inc was incorporated in early 2003. This means we were obviously up to other stuff than local – and we continue to.
As a quick example – I found value in domains, and picked up a few that I developed out. Some worked, some didn’t. Two top domains that were once owned by me were beat.com and webmaster.org. Both were developed out at one time – until a more lucrative offer (or better opportunity) made their sale an easy decision.
Local to me is a long term project that will not be solved in 1 year, or even 5 years. It requires a seismic shift in thinking – in how the YP business operates, in how consumers perform local search engines, and in how small business owners treat and operate with the internet (clearly exemplified in my last post on people being jackasses). Right now, I feel like local search is a swimming pool. We were on the edge with our feet sticking in, and sites like Yelp.com and Menupages.com have pushed us fully into the water (albeit the shallow end). Product inventory and companies like Yipit and LDC are pushing us towards the deep end – but everything has to come together before we can finally enjoy and have fun in the deep end.
But of course – all of that will take time. And while as a company we are growing very well
in the local space (giving us the capital to expand in new directions – more on that next year), we have other areas that provide us with revenue too (being self-funded requires creativity). Occasionally I will point at one of our efforts – in fact, I had two posts I wanted to write about, but one of them I imagined would get incorrectly construed with local. And so this post to clear the air.
I spend maybe 90% of my thinking time towards local. That still leaves 10% to do some other interesting stuff.
- In: Management