[RTC List] local hosts - Why are we held hostage?

[Larry Goldberg]

I apologize up front that this is a long post, so if you're not  
interested in the topic of alternate broadband and local bandwidth  
issues, please pass on reading this.
For those of you who have been intimately involved in the local  
bandwidth discussion/, here's my 50 cents worth.

Some comments on the replies to my earlier post:

On Dec 28, 2008, at 12:38 AM, William Van Hefner wrote:
> 1. Keep in mind that any "Obama money" that might be coming is not  
> just going to appear out of thin air. The money HAS to come from  
> somewhere. Unless he knows something that no one else over the  
> course of U.S. history did, Obama has exactly two choices as to  
> where to get that money. He can either a.) Raise taxes; or b.)  
> Essentially tell the U.S. Treasury and Fed to fire up the printing  
> press and start printing a bunch of extra money to spend. In the  
> case of scenario A, any public funds raised through taxes will be at  
> the expense of the private sector. That being the case, you can  
> expect there to be a corresponding loss of available funds available  
> from private sources. In other words, private spending will simply  
> be replaced by government spending. Essentially, it's a wash. With  
> scenario B, printing extra money will simply raise the U.S. deficit  
> and directly result in inflation that will make all of our spending  
> money worth less. So, while jobs may be created and new services are  
> made available, they will inevitably cost much more than they cost  
> now. There is no money for nothing, or checks for free, I'm afraid.

I am trying to NOT make this a political debate therefore I'll try to  
tread carefully.  While I don't disagree that Obama money is not  
"free" let's deal with realities:

1) the Wall St. bailout was supposed to free up money in the market  
and it has NOT accomplished this goal so far.

2) the Wall St. bailout had less controls than anything that's being  
proposed for infrastructure and other investments.

3) the best expenditure for the Federal Gov't to do to stimulate the  
economy, in the opinion of many leading economists, is to invest in  
infrastructure which we all use which includes broadband and "last- 
mile' applications.

4) the process for determining whether to invest in infrastructure  
(and how much) will be far more transparent and involve more of our  
input than any Wall St. or Auto company bailout to date.

5) Since this is closer to a "done deal" than almost any other  
economic stimulus, we should get on board and help to direct the  
expenditure rather than complain about the fact that public money is  
being used to stimulate the economy.

> 2. Our "community leaders" are politicians. [snip]... As much as I  
> dislike getting involved in politics, we are going to need to take a  
> much more aggressive role in the future if we are ever going to get  
> our point across. It's like the principal of the"carrot or the  
> stick". At the moment, the RTC doesn't have a very big carrot and no  
> stick to speak of.

I disagree.  The RTC has been far more effective as a "public lobby"  
than almost any other group or individual in this community when it  
comes to technology issues.  The RTC has become a respected  
professional group which has had a major impact in many local  
technology issues.  We have been the "squeaking wheel" sufficiently to  
get the discussion into the public forum and through our tech beat  
articles I believe we have been a major stimulus for public tech  
education and advocacy.  We have influence - we just need to see it in  
the perspective of time.  I've lived here 30 years now and I can  
personally say that this area has changed a LOT over that time, but  
it's hard to see the change from year-to-year. Our economy is more  
integrated with the national economy than ever before (when we used to  
be almost totally dependent on the timber and fishing industries) and  
we have the Internet, IT and other technology to thank for much of  
this.  We just don't know our strength as an advocate and we're not  
always focused on what our priorities are, but I believe we are  
powerful and have a lot to say in how community leaders (politicians)  
communicate our message.

> I think that the single most important thing that anyone out there  
> could do to improve matters right now is to start throwing as much  
> business as they can to independent providers like 101NetLink. If  
> AT&T starts losing enough customers, they will eventually be forced  
> to take notice. Money talks, and its the pioneers like 101NetLink  
> that really deserve our support.


Amen to that!

On Dec 28, 2008, at 11:26 AM, Seer Snively wrote:
>> Ouch!!!  Say that the next time the fiber is cut (now happening  
>> about 2-3 times a year!) and you can't get money out of an ATM,  
>> can't use a credit card machine, can't dial long distance, or look  
>> for a _local_ company doing anything _local_ with the internet.  My  
>> last job, we got many free months out of our AT&T T1 because they  
>> couldn't stick to their 5 9's SLA (99.999% uptime).  That's only  
>> about 45 minutes a month down, and they were consistently worse  
>> than that.
>>

I appreciate the fact that AT&T (and SuddenLink) are doing a  
relatively poor job, but this doesn't justify the expense and  
resources required to do a second fiber line.  AT&T's (and by default  
SuddenLink's) problem is because they are too dependent on the fiber  
which is subject to disruption - we agree on this point.  This does  
not necessarily necessitate a second fiber, however.  If we had 3-5  
redundant microwave links (at a fraction of the cost of fiber for the  
whole package!) then we could do "load balancing" to shift load as  
necessary and route it wherever there was an available path.  That's  
the beauty of TCP/IP and we should utilize it to our benefit.   
Additionally, with local peering (which admittedly won't help the  
issue of local ATMs or cell phones) we would be able to stay in touch  
with each other within the region, even if the rest of the world was  
disrupted (fiber cut, fires, earthquake, etc...) The whole idea of  
peering (which most urban areas feature) is that bandwidth can be  
allocated based on demand and each peer can help with load shifting.   
In my opinion, our problem on the Northcoast is that we're totally  
dependent on one primary provider, NOT that we don't have enough  
bandwidth.

> About $350 per month for a T1.  Who else is paying more than  
> that!?!  Honestly, the only reason we got that T1 was because it  
> wasn't in an area served by DSL, which is currently getting me 5 Mbs  
> down, 1.5Mbs up for $35 a month!  Why would I _want_ to pay more?

I may not have been clear about this - but there are two issues:  a)  
consumer prices vs. service levels and b) wholesale prices to Internet  
resellers/ISPs. As far as consumer prices, this is a highly  
standardized price-point which is very competitive and service levels  
are often sacrificed to provide "cheap" service.  People don't  
understand that DSL and cable are "shared" services - that's why you  
pay more for a T-1 which is dedicated bandwidth.  Yes, SuddenLink and  
AT&T offers "up to 8 Mbps downloads" but it's shared, so it's going to  
vary depending on time-of-day, neighborhood usage and other bandwidth  
constraints - thus it dips sometimes to barely satisfactory levels at  
times.  To get the consistent levels of bandwidth which some people  
expect would cost $75 - 100/mo. and most people won't pay that.

The other issue is wholesale prices.  To operate a successful ISP in  
any area requires the ability to purchase a "link to the cloud" which  
is affordable.  As you know, when I ran our initial ISP we only had  
several T-1s at the time (which was sufficient 15 years ago), which is  
totally insufficient by today's standards.  To purchase a DS3 (45  
Mbps. - the equivalent of 30 T-1s) or an OC3 (154 Mbps - the  
equivalent of 103 T-1s) is prohibitively expensive in this market when  
compared with almost anywhere else (e.g. Redding or Santa Rosa).   
While a new fiber line would certainly help, it would be cheaper and,  
ultimately more secure, to have multiple microwave links on different  
paths which can all be done for a fraction of the cost of the fiber,  
which would also provide 2 additional benefits: a) increased bandwidth  
as needed (it's scalable, so it can be built as needed) and b)  
increased competition (which would help keep the consumer price  
down).  The trouble is that this is a small, rural market and we are  
demanding big-city services at big-city prices.  You can't have it all  
and that's always been the trade-off living here (or anywhere in rural  
America).

> Also, I have to agree with Kevin.  Many of my friends are running,  
> not walking away from Suddenlink because of amazing lag times (over  
> 1000ms on weekdays, and "good" at around 450ms on the weekends) and  
> unplayable gaming.

I'm not happy with SuddenLink either, but my point is the fact that  
people expect excellent service while demanding increasing bandwidth  
(how many "average" people are now streaming NetFlix and doing YouTube  
over their home networks?)  Given our population and the demographic  
income profile, I don't think people are willing to pay more for  
service that would support 2 fiber lines (but they could support  
multiple microwave path alternatives - such as 101NetLink).  If  
someone had a combined 101NetLink service with AT&T, they should have  
virtually 4/9 service).  Also, with many of today's microwave  
solutions, you can have <4 ms latency - plenty good for audio and  
video conferencing.  Unfortunately, at this time it requires  
subscribing to BOTH services, but with peering, that wouldn't be  
necessary.  Ask SuddenLink or AT&T about peering, however, and they'll  
laugh at you!  I'll let Seth tell you, if he choses to do so, about  
his experience trying to work with those guys.

>>  The commercial market in this area is very similar to a  
>> residential market - there are too few "heavy users" to justify the  
>> business case for the investment required to provide additional  
>> "alternate/redundant" paths.
> Um, I'm not sure if I believe this.  My PlayStation 3 never hits  
> more than 75kbs when playing games, and often much much less than  
> that.  Maybe about 100kbs when HOSTING games and I have 8 people  
> connected to me.

My point is that according to the RCC study (whether you believe their  
findings or not) they confirmed that business needs are little  
different from residential needs (typically e-mail, web browsing and  
some file-sharing, but little "heavy" usage like nationwide hosting or  
streaming services).  All the major IT players in this market (e.g.  
StreamGuys, local hosting services, and others) do their hosting out- 
of-the-area and HSU and CR have their own feeds independent of the  
business market.  In fact, at least according to the RCC study, there  
is an amazing number of companies throughout the region (4 counties)  
that still use dial-up and don't see much need to upgrade!  This  
supports my point - there is little need for a second fiber, other  
than for "back-up" - alternate paths can be built far cheaper and  
faster via non-fiber technologies.

> Now, Netflix streaming pegs almost 1.5Mbs (at best quality) for 2  
> hours, and using software like PlayOn, we can have 2 people watching  
> different movies at the same time.  Downloading the most recent  
> Ubuntu torrent might fill my pipe up with about 700kbs, but at that  
> speed a DVD iso only takes an hour or so.

More and more people are doing this today, but are they willing to pay  
more than $39.95/mo. for their broadband?  I think not.  People vote  
with their pocketbooks, not with their mouths.

> VoIP may only be about 60-70kbs a call, but video might be 5-7 times  
> that.  Of course, times that by how many people?
>
> There is a little chicken and the egg problem.  If we don't have a  
> redundant path, we will never have good local hosting, even for an  
> individual company to host their own email and external web server.   
> If we did get a 2nd fiber line, I know at least one company that  
> would get a T1 to every location. :-)

They can have this today without fiber.  Talk to 101NetLink - Seth  
does VoIP and I hear it's very successful.  I can see room in this  
market for more wireless ISPs which could provide additional alternate  
paths, but the business case is hard to make.  It would cost only a  
fraction of the cost (and headache) of fiber, but try raising money in  
this economy and you'll quickly see why it won't happen.  Imagine how  
much time and money and effort Seth has invested in his business and  
how long it took to grow and he has yet to fill his OC3 pipe!

>> 3) There are currently viable alternate paths ("redundancy") which  
>> exist and are locally supported.  101 NetLink is a local company  
>> (out of Garberville) which offers an independent (non-AT&T)  
>> microwave link to the Internet via Ukiah.
> This is wonderful, but if AT&T isn't peering with him, it's not  
> redundant unless you buy from both providers yourself!

That's exactly the problem.  If someone out there has $500k to invest,  
let's talk!  Other than that, stop complaining about it - AT&T and  
SuddenLink won't do anything about it on their own, so it's fruitless  
to complain.

>> It's not about bandwidth.  It's about redundancy and reliability.   
>> If we try to say it's about bandwidth, we will be sidetracked into  
>> your very valid points.
>>

My point exactly.  It doesn't require a fiber solution (at a cost of  
$5-10 million) to fix a problem like ours.  There is currently a  
company discussing bringing an alternate fiber path in the next year,  
but all I can tell is that it looks like more smoke & mirrors.  With  
today's tight credit markets, raising the capital necessary to build a  
new fiber path into a market like ours is going to be a hard sell in  
my opinion.  All I can say is that I'll believe it when I see it.

>> he lack of local investment money is a real challenge to developing  
>> local solutions.  I've raised considerable capital in the past, so  
>> I have first-hand experience in what it takes to raise money up  
>> here.  Until recently (!), investors wanted a rate of return  
>> commensurate with Wall St.
> Of course, Wall St isn't even giving Wall St returns right  
> now.  ;-)  Greed is a good servant, but makes a terrible  master.   
> I've always wondered why people wouldn't want to _build_ something  
> real instead of shuffling paper around on wall street.  I guess we  
> all found on recently.

Building a local solution sounds logical, but as any local business  
person can tell you, local investment capital is hard to find and  
expensive if you find it.  Virtually all our area's local capital (in  
the form of retirement accounts, stock investments, 401(k) accounts,  
etc...) goes outside this area, unless it's a publicly traded stock  
(e.g. PG&E, AT&T, BofA etc...)  You can complain all you want about  
the situation, but that's the truth and we have to live with it.  I  
commend all the local entrepreneurs who manage to make a go of it up  
here, but from my first-hand experience, it's very hard, very risky  
and very expensive to build an infrastructure-type business -  
especially if you have to depend on the major players as the "gate- 
keepers" to enter the market (try to compete with AT&T or SuddenLink -  
it's a challenge!)  There are fewer and fewer independent ISPs  
nationwide today and by the end of this recession there will be even  
fewer.  To be a player these days you have to have millions of dollars  
and huge markets, unlike the Northcoast.  I'm not saying that a small  
company can't make a go of it up here, but as you all know, it's damn  
hard to do.  The collapse of Wall St. may have proven a point that  
many of us had - that Wall St. was phony with unsupportable returns  
anyhow, but now the capital markets have frozen which in essence  
freezes everyone else out anyhow, so that's a new fact of life we all  
need to adjust to today.

> But thank you Larry, for your wonderful post.  You have a lot to say  
> on this issue, having worked in Humboldt County for so long in this  
> field.  Thanks!

Thanks to you too.  You've been in the "heart of the beast" so you  
know the challenges we all face.  We all need to be realistic in our  
expectations and try to accomplish what we can with the resources we  
have at hand.  While we need to look up to the horizon while we move  
forward, we cannot afford to trip on the rocks in the road directly  
ahead of us.  It's a balancing act and it's frustrating at times, but  
eventually we'll get there.