Gmail and the Electrical Grid: Looks the same

September 2nd, 2009

GMail had a large-scale cascading failure yesterday:

At about 12:30 pm Pacific a few of the request routers became overloaded and in effect told the rest of the system “stop sending us traffic, we’re too slow!”. This transferred the load onto the remaining request routers, causing a few more of them to also become overloaded, and within minutes nearly all of the request routers were overloaded.

This cascading failure sounds exactly like that other universal network we all share: The electrical grid.

Large blackouts are cascading failures compounded by the failure of the “fuses” meant to isolate still functioning parts of the grid from the failed part.

Sounds a lot like what happened to gmail.

Interestingly the smart people at Google have recognized exactly that:

we have concluded that request routers don’t have sufficient failure isolation (i.e. if there’s a problem in one datacenter, it shouldn’t affect servers in another datacenter) and do not degrade gracefully (e.g. if many request routers are overloaded simultaneously, they all should just get slower instead of refusing to accept traffic and shifting their load).

Sounds like a good start. But what happens if a datacenter catestrofically fails. The routers stop accepting requests because they are gone ( or can’t respond ). Prepare for the next GMail failure.

As wikipedia notes:

Modern power systems are designed to be resistant to this sort of cascading failure, but it may be unavoidable (see below). Moreover, since there is no short-term economic benefit to preventing rare large-scale failures, some observers have expressed concern that there is a tendency to erode the resilience of the network over time, which is only corrected after a major failure occurs. It has been claimed that reducing the likelihood of small outages only increases the likelihood of larger ones. In that case, the short-term economic benefit of keeping the individual customer happy increases the likelihood of large-scale blackouts.

Posted in technical | 3 Comments »

Tell Jerry McNerney why we need health care reform

August 21st, 2009

Help make the case for health care reform! Tell Jerry McNerney D-CA11 your story.

This is mine:

I am an entrepreneur starting my own company in Silicon Valley. I am reliant on expensive COBRA coverage for my own health care needs. COBRA runs out in a few months.

Without health insurance I will be forced to shutdown my company and my dream, and find another job building someone else’s dream.

I can’t afford to pay any health care for employees. So ironically I have to hire contractors from countries that do have universal health care. (or at least cheaper health care). Hiring anyone in the U.S. is too costly. Even if someone can work for minimum wage and equity, most software engineers will not do without health care insurance.

The best economic stimulus that Washington could enact is to take the economic burden of health care costs off the backs of small business and their employees. Enable people to realize their dreams without taking a chance on their health!

Not having to pay $13,000 – $15,000 / employee / year is a huge, huge, huge economic aid! For my own company this would have saved $40,000. This $40,000 could have been spent hiring people.

Jerry’s original email message:

Dear Patrick,

We’re in the midst of an historic debate on health care and closer than ever to enacting major reform.

Many of you participated in the health care survey I began circulating in April or in the telephone town hall on health care I held recently with almost 5,000 participants. Your thoughts and comments are appreciated and offer great insight.

Unfortunately, as you’ve probably heard, there are those in our country who want to block an open debate on health care. We shouldn’t lose the opportunity to have a productive and respectful conversation about the future of health care in this country.

I am not deterred by the current challenges or by those who seek to scare people into believing myths about the great changes we can make to the health care system.

I need your help to continue. We must stand together to create great change – our voices must rise above the din of misinformation.

Will you please sign my petition to show your support for health care reform?

I will continue to reach out to hear from you. Over the past month, I’ve traveled throughout the district to meet with small business owners, seniors, doctors and nurses. I’ve toured healthClick to watch video care facilities, including hospitals, clinics and local practices so that I can see our health care resources firsthand.

During my health care listening tour, I’ve heard again and again from people who are ready for change to our health care system. During these tough times no one should have the additional burden of worrying if they’ll have health care when their family needs it most.

I’m working hard to find a uniquely American solution to the problems of our current health care system. Every family should have access to high quality and affordable health care. We should crack down on insurance abuses such as preventing people with pre-existing conditions from accessing coverage. Every citizen should be able to choose the doctor they want to see and be free to make their own decisions on care for themselves and their families.

The fight isn’t an easy one. I am being attacked for my support of health care reform. I need your support during this crucial time.

Please take a moment to sign my petition, and if you’re able, consider a donation to the campaign so I have the resources to continue the fight.

The other side will use any means possible to continue their fear campaign against reform, including distortions, lies, and intimidation to stop us. We cannot allow this to happen.

We’re in this together.

Thank you for all that you do,
Congressman Jerry McNerney

Posted in political, social commentary, starting a company | No Comments »

Don’t let the lawyers run the business

August 17th, 2009

This past weekend, my sysadmin ( James Sparenberg ) and I, were figuring out which cloud hosting service to use. We had been pitched a number of times by GoGrid. I had been given a “try us out” credit by the very pleasant sales person. I was going through the process of signing up.

  1. name (check)
  2. company (check)
  3. address (check)
  4. read the Acceptable Use Policy, Beta Agreement and the Terms of Service… uh, oh

Beta Agreement:

2.  You will not disclose any Confidential Information to a third party, including without limitation a GoGrid Competitor, or use it for any purpose other than to facilitate beta testing.  However, you may disclose Confidential Information to the extent required by law, provided you give GoGrid advanced notice reasonably sufficient to allow it to contest such disclosure.  “Confidential Information” refers to any information regarding the Service unless such information is: (a) provided at the GoGrid Website (http://www.gogrid.com) and made available to Internet users without an account or password; (b) already publicly known other than through your act or omission; or (c) made available by GoGrid to customers of the Service after beta testing and after the official public launch of the Service.

3.  You agree that violation of the provisions of this Beta Agreement might cause GoGrid irreparable injury, for which monetary damages would not provide adequate compensation, and that in addition to any other remedies available, GoGrid will be entitled to injunctive relief against such breach or threatened breach, without the necessity of proving actual damages.

Danger, Will Robinson, Danger. At GoGrid’s sole discretion, they can go after me for lots of money and high-legal fees. According to a strict reading of this agreement, my balance and usage information is GoGrid’s confidential information.

It gets worse with GoGrid’s Acceptable Use Policy

A. The following activities are expressly prohibited:
2. Intellectual property infringement, including violations of copyright, trademark, and patent rights, and use or distribution of pirated software.

B. Disruptions & security:
GoGrid may suspend Service in whole or in part if it reasonably suspects an AUP violation. Customer will reimburse GoGrid for any expenses resulting from Customer’s violation of the AUP, including attorneys’ fees. GoGrid may also disable Customer’s service if GoGrid suspects that such service is the target of an attack or in any way interferes with services provided to other customers, even if Customer is not at fault. GoGrid does not issue refunds for terminating service due to any of the causes above.

So if GoGrid gets any sort of DMCA notice, legitimate or not, GoGrid can decide to take company’s website offline without compensation. If an Amplafi user is abusing the service, GoGrid will shutdown our entire service.

This arbitrary exposure to business disruption is unacceptable. If any corporate officer agreed to these terms I would fire them.

GoGrid’s (unacceptable) Terms of Service:

4. Acceptable Use.

(ii) Notwithstanding any provision to the contrary in this Agreement, and without limiting any of GoGrid’s rights or remedies, GoGrid may suspend Service in whole or in part in the event that GoGrid reasonably suspects an AUP violation. Reasonable suspicion pursuant to the preceding sentence includes, without limitation, a third party notice or claim that Customer’s use of the Service infringes on third party rights. GoGrid will make reasonable efforts to notify Customer before any such suspension, unless the AUP violation calls for immediate action to prevent injury or liability, in GoGrid’s opinion and at its sole discretion. Suspension pursuant to this Subsection 4(a)(ii) may continue so long as GoGrid reasonably suspects an AUP violation. GoGrid is not liable for any Service suspension authorized by this Subsection 4(a)(ii), or for any related loss, even if the suspected AUP violation did not occur.

GoGrid will shutdown Amplafi’s website for any reason at all. “Reasonable effort to notify” is not defined and is highly subject to interpretation.

6. Maintenance & Security.
GoGrid is not responsible for providing physical access to or copies of software, data, or content stored on GoGrid’s equipment under any circumstances and is not required to provide network access (i) after any termination or suspension of Customer’s Service or (ii) in the event of hardware failure, abuse by hackers or other third parties, improper administration by Customer, or other interruption of network access.

GoGrid will shutdown an account for arbitrary reasons and then discard all customer data…. if this is a day that ends in a ‘y’.

8. Warranties, Disclaimers, & Limitations of Liability.
(b) GOGRID WILL NOT BE LIABLE FOR ANY CONSEQUENTIAL, INCIDENTAL, INDIRECT, EXEMPLARY, PUNITIVE, OR MULTIPLE DAMAGES, EVEN IF ADVISED IN ADVANCE OF THE POSSIBILITY OF SUCH DAMAGES. GOGRID’S MAXIMUM LIABILITY ARISING OUT OF OR RELATED TO THIS AGREEMENT WILL NOT EXCEED THE TOTAL AMOUNT OF FEES PAID BY CUSTOMER DURING THE 12 MONTHS PRECEDING THE INJURY GIVING RISE TO THE CLAIM.

And you can’t do squat about it.

Sorry! No sale!

We decided to go with Rackspace.

Rackspace (reasonable!) Terms of Service

7. Law/AUP. You agree to use the Services in compliance with applicable law and our AUP, which is incorporated by reference in the Terms of Service. You agree that Rackspace may, in its reasonable commercial judgment consistent with industry standards, amend the AUP from time to time to further detail or describe reasonable restrictions and conditions on your use of the Services. Amendments to the AUP are effective on the earlier of our notice to you that an amendment has been made, or the first day of the next Renewal Term. You agree to cooperate with our reasonable investigation of any suspected violation of the AUP. In the event of a dispute between the parties regarding interpretation of the AUP, our commercially reasonable interpretation of the AUP shall prevail.

Wow! A ToS requires that the Customer be proactively notified!

8. Your Information. You represent and warrant to Rackspace that (i) all information you provide to Rackspace for purposes of establishing and maintaining the Services is accurate; (ii) if you are an individual, you are at least eighteen years of age; (iii) you will not use the Services for the development, design, manufacture, production, stockpiling, or use of nuclear, chemical or biological weapons, weapons of mass destruction, or missiles in any country listed in Country Groups D:4 and D:3 of Supplement No. 1 to Part 740 of the United States Export Administration Regulations, and (iv) you will not provide access to the Services to any person (including a natural person or government or private entity) located in or a national of embargoed or highly restricted country under United States Export Regulations, which include as of June, 2008, Cuba, Iran, Iraq, Libya, North Korea, Sudan, or Syria. You agree that Rackspace may, without notice and without liability to you report to the appropriate governmental authorities any conduct by you or any of your EUs that Rackspace reasonably believes violates applicable law, and provide any information that it has about you and your EUs in response to a formal or informal request from a law enforcement or government agency or in response to a formal request in a civil action that on its face meets the requirements for such a request.

Notice the last line, the request must be official — not just some sort of automated DMCA notice generated by a spambot in Hollywood.

12. Suspension/Termination.
(a) Suspension of Services. You agree that Rackspace may suspend the Services if: (i) Rackspace reasonably believes that the Services are being used in violation of the AUP; (ii) you fail to cooperate with any reasonable investigation of any suspected violation of the AUP; (iii) Rackspace reasonably believes that suspension of the Services is necessary to protect its network or its other customers, (iv) as required by a law enforcement or government agency, or (v) if the Card cannot be charged for payment in accordance with Section 5. You agree to pay a reasonable fee for reinstatement (“Reinstatement Fee”) following any suspension.
(b) Termination by You. The Terms of Service may be terminated by you at any time as long as all Fees then due together with unpaid Recurring Fees for the remainder of the Initial Term or the Renewal Term, as the case may be, are fully paid on the business day following the termination date.
(c) Termination by Rackspace. The Terms of Service may be terminated by Rackspace prior to the expiration of the Initial Term or any Renewal Term without liability as follows: (i) upon seventy-two (72) hours notice if you are overdue on the payment of any Fee; (ii) you materially violate any provision of the Terms of Service or the AUP, and fail to cure the violation within ten (10) days after receipt of a written notice from Rackspace describing the violation in reasonable detail in our sole discretion; (iii) upon twenty-four (24) hours notice if the Services are used in violation of a material term of the AUP more than once, or (iv) upon twenty-four (24) hours notice if you violate Section 8 (Your Information).

Notice the explicit difference between GoGrid’s termination policy and Rackspace. Rackspace says that they will suspend unilaterally. GoGrid goes right to termination. Rackspace explicitly lists out timeframes. Rackspace imposes a 10-day advanced written notification requirement upon themselves. Furthermore, Rackspace requires that the violation be material ( i.e. significant ) and repeated.

14. Confidential Information.
Information that is developed by a party on its own, without reference to the other’s Confidential Information, or that becomes available to a party other than through violation of these Terms of Service or applicable law, shall not be “Confidential Information” of the other party. Each party agrees not to use the other’s Confidential Information except in connection with the performance or use of the Services, as applicable, the exercise of its legal rights under these Terms of Service or the Order Form, or as may be required by law. Each party agrees not to disclose the other party’s Confidential Information to any third person except as follows: to its respective service providers, agents and representatives, provided that such service providers, agents or representatives agree to confidentiality measures that are at least as stringent as those stated in these Terms of Service; to law enforcement or government agency if requested, or if a party reasonably believes that the other party’s conduct may violate applicable criminal law; as required by law;
or in response to a subpoena or other compulsory legal process, provided that the disclosing party must give the other party written notice of at least seven days prior to disclosing Confidential Information under this subsection (or prompt notice in advance of disclosure, if seven days advance notice is not reasonably feasible), unless the law forbids such notice.

Wow, once again a reasonable time to hire legal talent to address a legal issue.

Rackspace Acceptable Use Policy:

Copyrighted Material
You may not use the Rackspace Cloud’s network or Services to download, publish, distribute, or otherwise copy or use in any manner any text, music, software, art, image or other work protected by copyright law unless:
• you have been expressly authorized by the owner of the copyright for the work to copy the work in that manner; or
• you are otherwise permitted by established copyright law to copy the work in that manner.
It is the Rackspace Cloud’s policy to terminate in appropriate circumstances the services of customers who are repeat infringers.

What a difference the lawyers can make! GoGrid’s ToS, AUP, and Beta agreement are completely one sided and read like some free consumer service, not something that should be entrusted with any serious business. Rackspace’s agreement is balanced. Gives everyone an opportunity to seek legal advice. And more importantly, treats the cloud services as running serious business applications.

Its worth noting that under GoGrid’s AUP, ToS and Beta agreement — Facebook, YouTube, and many other popular legitimate services would be shutdown.

No thanks, GoGrid.

Posted in amplafi, broken, starting a company | No Comments »

Blame. NO. Responsibility. YES!

August 16th, 2009

When reaching for the stars, something will go wrong.

Rockets blow-up.
Servers crash.
Regressions happen.

How you handle the setbacks is critical. Blame is a useless response. Blame is negative. After blame has been assigned, the rocket is still in pieces, the server is still down, and the bug still exists.

Hire people that take RESPONSIBILITY for finding SOLUTIONS. Hire people that look to HELP others shoulder the RESPONSIBILITY to fire the problem. Hire people that look for ways to prevent a duplicate of the same problem.

Once the problem is fixed, do you and your company spend time praising the “firefighters” only? Do you spend any time praising the person who caused the fire but was RESPONSIBLE enough to step forward, admit the problem, and help fix it?

Do you take RESPONSIBILITY as a manager to give your people time to build a “sprinkler system” to put out a similar future fire?

The first “fire” might be caused by someone else’s carelessness. But the second fire is YOUR responsibility if you didn’t budget time and money for that “sprinkler system”.

Posted in amplafi, management, starting a company | No Comments »

Mac OSX — the most insecure OS around

August 7th, 2009

A year ago, I gave up on Windows. I was tired of the nervous eggshell feeling with anti-virus software, security patches, and a machine that would mysteriously be slower and slower no matter what I did.

Because of MacOSX Unix roots, I made the switch to MacOSX confident that my personal computer would be safe.

This faith has been severely shaken. I now regard MacOSX as one of the most INSECURE operation systems.

MacOSX has a dangerous default DNS/DHCP configuration. Even worse, this dangerous configuration can not be fixed from the UI. Even the command line fix is difficult. And worst of all Apple is aware of this and does nothing.

There are 5 bits of background you should know:

  1. DNS is a fundamental part of the internet. DNS is the ‘name resolution’ service that converts ‘mail.google.com’ into the ipaddress: ‘74.125.19.19′ which is what your computer really uses to contact GMail servers so you can read your email. This conversion from the human readable ‘mail.google.com’ to ‘74.125.19.19′ is analogous to the post office converting the postal address on your snail mail envelope to a Zip+4 encoding that is printed at the bottom of the envelope. This encoding is what is actually read by the postal service mail sorting machines to determine where your snail mail goes. Now imagine that the postal service’s encoding machine was compromised. This compromised postal encoding machine was changed so that no matter what the Zip+4 code was supposed to be the machine always encoded the location of Dick Cheney’s house. As a result, all your mail that was processed by that compromised postal service would go first to Dick Cheney. Dick Cheney would get a chance to open all your snail mail, read it, copy it, etc. He could then reseal the original envelope, reencode the envelope with the correct barcode and put the envelope back in the postal service system to be delivered to the correct address. so that no matter what address you had actually printed on your envelope. All this would happen without you being aware of the problem nor able to stop it from happening.

    Everything internet related depends on correct translation of ‘apple.com’ or ‘bankofamerica.com’ to the correct IP address, not some third party server. How does the your machine know that it is communicating with apple.com and not some evil server? Your computer relies on the DNS lookup being correct. If the DNS lookup is compromised then when your software update runs to check for the latest security patches it is really installing a virus from evilserver.com not apple.com.

    There has been recent concern about DNS spoofing. ( links ). Being the cautious person that I am, I decided to explicitly listed opendns.org’s DNS servers (208.67.222.222 and 208.67.220.220) as the DNS Servers to trust in my Network configuration. I felt pretty cocky and safe.

  2. The second bit of the puzzle is DHCP. In order to talk to the world, computers need to have their own personal unique ipaddress (it’s very own ZIP+4 code). Every time you go into an internet cafe and pop open your laptop, your laptop uses the DHCP service to figure out what unique ipaddress (192.168.1.101) it should use while you are in that cafe. DHCP is nice because otherwise you would have to manually figure out and set an ipaddress for your computer that is different than everyone else’s laptop. And if someone else picks the same address as you did, all of a sudden your internet connection starts behaving odd. In addition to supplying a ipaddress, the DHCP server also supplies a DNS server that should be used. This is useful when you have your laptop at work and you need to go to an internal website such as http://go/wave Notice there is no ‘.org’, or ‘.com’ after ‘http://go/’ this means that ‘go’ is only visible when you are at work and can access the internal DNS server using the information that the corporate DHCP server supplied to your laptop.

    So to summarize DHCP supplies your laptop with the information needed for the world to talk to your laptop ( by assigning an ipaddress to your computer) , and helps you find out about the world (by telling your computer about the corporate DNS server). DNS servers enables your laptop talk to the world by giving your laptop a ‘go-to-machine’ for all its addressing questions.

  3. Third, DNS servers are usually big expensive computers secured by “smart people”. However, the DHCP server is really just a bit of software running on a Linksys router at your internet cafe. Your internet cafe’s Linksys router probably has the default password and no one ever checks on it. There are millions of this routers, with minimal security, no one checking on them and your laptop is trusting these unsecured routers with the keys to your kingdom. Your laptop is asking this router — “tell me which DNS server to trust?”
  4. Fourth, Lets say that you are paranoid enough to say ” ohh this is bad. I am not going to trust such a router in a greasy, dark corner with telling my precious laptop which DNS server to trust.” So if you are like me you configure your laptop with an explicit list of DNS servers thinking that your laptop, especially your oh-so-secure Mac would never disobey you about something so critical as DNS.
  5. Fifth, you would be wrong. The insecure MacOSX does disobey and it does trust that greasy spoon router over you. MacOsX doesn’t let the user (YOU!) say that only certain machines are allowed to be your laptop’s DNS servers! Furthermore even if you have supplied your own custom DNS servers that you trust, the insecure MacOSX trusts the greasy spoon DHCP server’s DNS servers over your trusted DNS servers. And there is NO way to convince MacOSX otherwise.

This means that if the DHCP server at your internet cafe has been compromised you are as well.

How I found out

So here I am feeling all cocky and safe. I type in my company’s web address, ‘amplafi.net’ and amplafi.net resolved to 113.29.236.168 which offered that the website was for sale!… I freaked out!

hacked-dns

I discovered my MacOSX laptop was insisting on trusting these EVIL DNS servers 206.13.28.12 206.13.31.12 . Was the 10.5.8 OSX patch that was installed 6 hours ago really what it seemed? Who knows? When I installed updates to Firefox plugins was I really installing the correct versions or a compromised version that would report back to some site in Russia all my bank account information? I have no way of knowing.

This is the really scary part about everything. John Simpson reports:

Under 10.4 and earlier, when I specified a custom nameserver, the system would use only the nameserver(s) I specified. However, under 10.5 Apple has apparently changed that behavior, and uses my specified nameservers in addition to the DNS servers specified by the DHCP server. It shows the DHCP-provided server IP on the list, greyed out, so you can’t delete it.

For a while, I adopted a “grin and bear it” attitude — after all, the DHCP server at home is handing out the IP of my internal Linux server (also running djbdns) as the DNS server, so I was only unsafe when I used the laptop outside the house. However, with the recently announced vulnerability in the DNS protocol, the massive world-wide patch effort by major DNS vendors, and the fact that many networks haven’t applied the patches yet, I don’t really feel safe relying on anybody else’s nameservers.

I tried calling Apple about this, but it turns out that my AppleCare contract doesn’t cover technical support such as this.

My next approach was to just brute-force search the system for anything relating to DHCP. It took a while, but I was able to find the file which needed to be changed, and figure out the necessary changes. Basically, I found a file which controls which options are used by the DHCP client when handling a response from a DHCP server. I removed the DNS-related options from this list, and after rebooting the system, the laptop now ignores the DNS server options being sent by the DHCP server.

The file I found is named IPConfiguration.xml, and it’s buried in this folder: /System/Library/SystemConfiguration/IPConfiguration.bundle/Contents/Resources. You need to create a copy of that file, edit the copy, and remove a few entries in the DHCPRequestedParameterList key. (The entries to remove are those for 6, 15, and 119.) I have added full details on this process to my djbdns setup page, in the section titled Disabling DNS servers from DHCP.

Thankfully I found this web page from 2008(!) that showed how to fix this problem (thanks John M. Simpson):

It is possible to make the DHCP client ignore the “DNS server” options in the DHCP response. It’s not for the faint of heart, but if you’ve been able to handle the rest of the instructions on this page, you can handle this bit as well.

I have done this on my own laptop (a MacBook Pro) and it does work.

Be aware that this is a GLOBAL change. If you do this, your machine will not use the DNS servers specified by any DHCP server. This may affect your machine’s ability to easily work with corporate networks (especially those using Windows Active Directory) or other networks which use private DNS namespaces.

$ sudo -s
Password: You will not see your password as you enter it.
# cd /System/Library/SystemConfiguration/IPConfiguration.bundle/Contents/Resources
# vi IPConfiguration.xml

Find this block...

        <key>DHCPRequestedParameterList</key>
        <array>
                <integer>1</integer>
                <integer>3</integer>
                <integer>6</integer>
                <integer>15</integer>
                <integer>119</integer>
                <integer>95</integer>
                <integer>252</integer>
                <integer>44</integer>
                <integer>46</integer>
                <integer>47</integer>
        </array>

Comment out the 6, 15, and 119 entries. The result should look like this:

        <key>DHCPRequestedParameterList</key>
        <array>
                <integer>1</integer>
                <integer>3</integer>
                <!-- commented out so that Bad DNS servers coming from DHCP servers
                are not used.
                <integer>6</integer>
                <integer>15</integer>
                <integer>119</integer>
                -->
                <integer>95</integer>
                <integer>252</integer>
                <integer>44</integer>
                <integer>46</integer>
                <integer>47</integer>
        </array>

Save your changes.

Be sure to flush the DNS cache.

On MacOsX:

dscacheutil -flushcache

I have tried John’s suggestion and those scary DNS servers are no longer present. But has my machine been compromised already? I will be visiting the apple store in a few hours asking for answers.

The story continues. I was sitting next to a customer. On her windows box she was picking up the same bad DNS servers. It wasn’t until later when I got home that I discovered all this information. I suspect (but am not completely certain) that windows will not override an explicitly specified DNS server.

Update: So after talking with some people, its pretty clear that MacOsX shares this issue with Windows XP because offering out internal DNS servers is part of what DHCP does. However with Windows XP, it is easy to explicitly lock down the DNS servers.

How to lock down a Windows XP box with safe DNS servers:

On Windows:

network-connections-step1

before-changing-dns-step2

use-opendns-dns-servers-step3

To clear Windows DNS cache:

ipconfig /flushdns

Posted in broken, how to, technical | 1 Comment »

The 100-hour work week myth

July 5th, 2009

Chris Yeh calls out workaholism as the stupid choice it is:

If you work 100-hour weeks, no one (investors, co-founders, employees) can blame you if things don’t work out, right?

And I like to think I’ve worked a lot smarter since then [missing dinner with spouse].

The life of an entrepreneur can be rough, but at least it’s a life of your choosing. The same can’t be said for your family. Give then a chance to make their own choice.

In other words, it is the default choice in the valley and in the technology sector. And its a stupid choice. 168 hours in the week. 100 hours at work. Allow 8 hours/day for sleep. Drive-time to/from work of 1 hour. This leaves exactly 13 hours for the employee to do *anything else*.

A few years ago, I had a job with the best work-life balance. This start-up had with only 7 engineers with 30-ish total people. Between November and January, we built a Paypal integration and a major piece of functionality that got the start-up their first bits of solid revenue. Everyone took their normal holiday vacation. Every programmer worked 9-5. No weekend work. We completed the project on-time.

The company is LinkedIn. We achieved this because Jean-Luc Vaillant was fanatically about knowing exactly what was to be built and automated tests so he knew exactly where the code was. Those tests had to pass each and every night. No new work was to be done until all the previous night’s failed tests were fixed.

Every later employer had to live up to this reasonable bar. Sadly most fail and most projects are late.

They fail because the managers listen to the siren song singing the lies:

  • that says that automatic tests are optional;
  • “trusting” the developer to adequately test by hand is good enough;
  • that there is more time to do-it-again than to do it right
  • that documentation is optional and it better to have team members figure out anothers work than it is to demand that the creator document;
  • and that long hours are better than sane hours

While Chris does touch on the work-life balance with his wife, he misses some key points. If the team is working 100-hours/week:

  • the team has no reserve capacity – if a short-term sprint is needed to wrap up a project – forget it
  • the team starts to waste time at work: web surfing and game-playing. So while physically there, they are neither productive nor getting a break from the work environment.
  • as soon as there is any corporate setback – moral collapses. When it looks like the company is going to be the next Google, employees will justify to themselves that working ridiculous hours will pay-off. This illusion is dispelled at the first severe setback.
  • someone outside of work is always telling the employee how stupid they are to work such long hours. The wife, the husband, the kids, the mother, or just the friends who are going up for that most excellent ski trip to Lake Tahoe.

So my advice to employers:

  • Get rid of the game room. Make employees have fun outside of the building.
  • Cut the power to the employees computers at midnight. Make them sleep so they can think and not make silly mistakes.
  • Do a postmortem on every crisis. Without blame and with automation ONLY, look for ways to make sure that the crisis can never, ever repeat. Working “harder” or requiring greater “perfection” is NOT the answer.
  • Reward employees – not for working harder, freeing up ‘capacity’. Did some developer, IT person, or janitor do something or automate something that freed up 20 minutes/person/week? In a 30-person startup, those 20 minutes saved is the same as hiring a full-time person for 3 months! Get everyone to look for these “small” time-savers. Work now becomes less onerous, more enjoyable, and your headcount stays down.

Expanding on the last point with some examples:

  • Automatic tests — avoids developers acting like monkeys do manual tests.
  • Buy the absolute fastest machines. My latest machine took me from 15 minutes builds to 1m30second builds. I started running the tests all the time!
  • Virtual assistants to handle the random shit that an employee might have to do during the day
  • Every 6 weeks, a mobile oil change service so that no one needs to run to Jiffy Lube
  • Outsourcing human resource issues

Spend the time to discover those “small, annoying” things that seem to petty to complain about — but that impact a significant percentage of the company.

Remember for a small 30-person startup saving 1hr20m/person/week ( i.e. 16min/person/day ) is the same as hiring another person. And in the process, enables everyone to step back from the brink.

Google has their famous 20% “free” time to work on new projects. Every startup should have 20% “free-up” time to make existing projects less painful.

While I am working hard at amplafi I am working even hard on making sure that my family knows I much rather be with them than coding.

Also read Steve Blank’s post on the Lies told Entrepreneurs.

Update ( 27 July 2009 ) My response to Paul Jozefak, a German VC, guest blog post:

Strongly, strongly agree with:

Ask me what I see lacking most in startups in Europe and I’ll say hunger, drive, and lofty goals.

For me my hunger and drive come directly from wanting to change the world for my children.

So I equally strongly DISagree with:

worked four jobs for the money to launch their venture, without giving a second thought to “quality of life” or “spending time with the kids.”

For me sacrificing the hours between 6:30-9:30pm that I spend with my kids is a false choice. I sacrifice that time only when absolutely necessary and never more than 2 days in a row. Once I have those 3 hours with family, I am emotionally recharged and able to focus completely on building my company, Amplafi.

I am not alone in this. Chris Yeh and Steve Blank : Lies Entrepreneurs Tell Themselves share my feelings.

My personal reality is the least successful company demanded the worse and longest hours. And the most successful startup asked the most reasonable hours. We work from 9-5. No weekends. No missed holidays. You might have hear of it. Its called Jean-Luc Vaillant did his job and managed his people well.

Shitty long hours is not a badge of honor. Its a sign of bad prioritization and resource management. Sure some times the long hours are necessary… just like a sprint is necessary at the end of a marathon. But you don’t sprint the entire length of the marathon. And unlike a marathon in a startup, there is no rest after crossing the first finish line – just another finish line in the distance.

A startup that is sprinting constantly better hope that they get bought before exhaustion sets in. Otherwise their competitors that have paced themselves better will pass them up and their best people will burned out and quit. Any little stumble, any sign that success and glory are a few months away… and the startup starts spending time looking for fresh blood.

Posted in amplafi, management, rants, starting a company | 4 Comments »

Open Letter to Virtual currency companies: “universal” is not a feature

July 3rd, 2009

Interesting post from Lisa Rutherford about reaching for a universal currency and some discussion about some problems with that dream.

While the glories of a “universal” currency are appealing, I think this might be a “feature” that is in fact a negative.

First some questions:

  1. Don’t we already have a “universal” virtual currency called the US dollar (and the Euro)?
  2. Europe has been working very hard at the euro. Struggling with dissimilar economies that are only beginning to work together. Some countries had this tendency to spend to solve problems (Italy, Greece, Spain). Others were more conservative in their money printing philosophy (Germany). Working through these issues has been a constant source of tension. How will this be any easier between two different companies with different philosophies about how virtual currencies should be used?

Virtual currency companies should look at casinos and the collectible market instead.

Casinos issue casino chips for very good reasons. If gamblers use bank notes to place their bets then every bet becomes a purchasing decision: “I could place this $20 bill on Red 7, or I could buy a steak dinner”. Chips makes the purchasing decision happen only once. Redeeming chips has a “cost” — the gambler has to find the cashier. The cashier is not near an exit. The gambler then still has to escape the casino with the cash resisting temptation all the way.

Casinos also issue special chips that cannot be redeemed. These chips are billed as “Your first bet is free” chips.

Lastly, some casinos use chips as a branding, souvenir opportunity. A percentage of chips are never exchanged representing free money to the casinos.

Because casinos allow exchange out of their “virtual currency”, they have to spend a lot of time and effort on complying with money laundering regulations. By striving for universality, virtual currency companies will subject themselves to the same regulations.

Virtual currency companies should instead serve the same purpose as casino “first-bet” chips. Non-redeemable, can only be used to have fun, and to not make it obvious to the consumer that they are spending money.

A universal “Linden dollar” or “Lisa dollar” looks and feels too much like a “real” dollar to pay the real rent. The “currency” should stick to “toy/game-like” characteristics: “magic dust”, “gold”.

Virtual currency companies should steer away from “purchasing” words to “barter” words: “trade”, “exchange”, “collect”.

Additionally look at trading card companies like Topps, Upper Deck, and Magic: The Gathering. Very arguably these companies have been profitably exchanging unwanted dollars for valued cardboard for years. Trading cards were a virtual currency long before “virtual currency” was a buzz word. I do know that goplaynetwork.com is working on such a system.

Collectability is the direction that virtual currency companies should head toward — not universality.

Greg Berry also commented on Lisa’s Venture Beat virtual currency post. He touches some of the same themes as this post but he focuses more on the social aspects of virtual currency. He refers to : tuggl , twollars , openmoney and cyclos.

Greg Berry is correct. The social aspects of virtual currency need to be enhanced not the universalness

Posted in marketing, starting a company | 2 Comments »

Self-sacrifice does not always come from a bullet

June 26th, 2009

From an email:

On Tuesday at 8 a.m., I will stand trial for speaking three truthful words: “I am gay.”

On Tuesday, I will face a panel of colonels who will decide whether or not to fire me — to discharge me for “moral and professional dereliction” under the military’s “Don’t Ask, Don’t Tell” policy.

On Tuesday, I will try to prove that it’s not immoral to tell the truth.

As an infantry officer, an Iraq combat veteran and a West Point graduate with a degree in Arabic, I refuse to lie to my commanders. I refuse to lie to my peers. I refuse to lie to my subordinates.

My case requires that I provide personal testimony from people who can attest to my character. That’s why several members of my military unit have written letters of support and offered to testify on my behalf.

Now I need your help. ANYONE who believes the Army should not fire me can take a stand right now. I am bringing a statement of support to Tuesday’s trial and I need you to add your signature to it. Will you support me by signing this statement before Tuesday?

I want to thank the 141,262 people who have signed the “Don’t Fire Dan” letter launched a few weeks ago by the Courage Campaign and CREDO Mobile to President Obama, asking him to take leadership to bring this tragic policy to an end.

The momentum is building. This week, 77 members of Congress signed a letter to the President citing my service as an example of why DADT should be repealed. And a Gallup poll was recently released showing that 69 percent of Americans — including 58 percent of Republicans – favor allowing openly gay men and lesbian women to serve their country .

As I learned at West Point, deception and lies poison a unit and cripple a fighting force. That’s why more than 70 of my fellow West Point graduates have also come out of the closet to join Knights Out, the organization I co-founded to build support for the repeal of “Don’t Ask, Don’t Tell”.

The only way we will eventually overturn “Don’t Ask, Don’t Tell” is by speaking up together. You can help me fight back right now by adding your name to my statement of support. On Tuesday morning, I will bring your signature — and thousands of others — to my trial as a demonstration of your collective support:

http://www.couragecampaign.org/SupportDan

National security means many things, but the thing that makes us secure in our nation and homes is love. What makes me a better soldier, leader, Christian and human being is love. And I’m not going to hide my love.

Love is worth it.

Thank you for your support.

Daniel W. Choi
1LT, IN
New York Army National Guard

And my response:

We should treat honorably servicing members of the military with honor.

Court martialing Lt. Choi is dishonoring his service. Court martialing Lt. Choi will stain the Army not Lt. Choi.

Court martialing Lt. Choi clearly indicates that for the U.S military, the words “honor” and “dignity” should be prefaced with “mostly” and “while convenient”.

Being willing to face a court martial in order to do the honorable action is the highest indicator of honor that any service member could demonstrate. This willingness to sacrifice oneself is what the military demands. Self-sacrifice doesn’t always come in the form of a bullet.

The hardest form of self-sacrifice is willingness to be subject to societal rejection.

Lt. Choi should be promoted not court martialed.

Posted in political | No Comments »

Open Message to the anti-tax crowd: move

June 17th, 2009

Amazon’s definition of “unconstitutional” :

“We don’t like it”

I love all these people who whine about taxes.

Don’t like taxes? Move to Somalia. No functioning government since Bush the First – a libertarian paradise.

The anti-tax people complain about taxes but want the benefits of:

  • a functioning University system so that there are high-quality people to work at your cool start-up
  • a public school system that at the very least keeps kids off the street. (50% of California’s budget)
  • roads
  • police
  • prisons ( 10% of California’s budget )
  • state parks
  • a functioning emergency system for the next airline crash or the next earthquake
  • airports
  • Caltrain
  • weights and measures people to make sure that when you buy a gallon of gas you get your full gallon
  • code enforcement to make sure a restaurant is not serving 3 month-old rotten meat and the kitchen is not infested with cockroaches
  • zoning enforcement that stops your neighbor from running a chicken farm
  • labor laws that stop child labor and insist that your employer actually has to *pay* you
  • SEC laws that require that companies follow GAAP
  • laws that allow lawsuits and action against companies when they pollute the water you drink

Next time you think that the government does nothing for you. Spend some time finding out how badly mainland China, Dubai, or Yemen allows the powerful to abuse everyone else.

I can go on and on.

But seriously, grow up. You want to live in a civilized society? expect to pay for it — it does not come for free.

Oh sure, it isn’t perfect — fine make it better.

As for me, I am happy to pay taxes and enjoy the best state (California) in the US. There is no way I would move to another state that has worse laws.

Update 1:

A few other “unnecessary services” from the government:

Move to Somalia. Avoid these annoying bureaucrats!

Meanwhile think about this:

  1. In the 70’s corporations paid 2/3 of the taxes, today after a full generation of the rich whining about taxes corporations pay 1/3 of the taxes.
  2. 30 years after Prop 13, the biggest beneficiaries of Prop 13 are corporations because they never, never sell property ( 99-year leases anyone? ).

All this anti-tax rhetoric has allowed corporations to shift the tax purden to the individuals.

Maybe it is time to wake-up about this scam and stop buying into the anti-tax rhetoric so blindly.

Update 2:
More from TechCrunch:

State governments assess “emergency” tax measures to get quick money because they can’t bear the thought of making the tough choices necessary to cut spending

Hmmm… so in hard economic times, when a social safety net is that much more important you want states to cut funding.

So from your perspective:

  • Unemployment insurance
  • Job retraining programs
  • Community Colleges
  • 4-year public universities
  • subsidized day care
  • subsidized elder care
  • Section 8 housing assistance
  • School lunch programs
  • morning/afternoon pre-/post- school day programs
  • community grants for starting a new business

Should all be cut.

Someone who has just been laid off can try to job hunt and compete with 500 applicants hoping they can keep their head above water economically;

Or

they can return to school and complete their AA degree or the BS, or get their MBA using the above listed services to make it economically possible.

By providing these services, a state enables their citizens to be more valuable and more productive when the economy turns around.

The single mom ( or dad ) may start the downturn with no college education. Through the services listed above, this single parent could end up with a degree that will enable them to double their income. Or give them the skills to start their own business.

Yet you make the interesting choice that states should remove this opportunity to turn economic lemons into lemonade.

Interesting. I am curious why you think that a less-educated workforce is a good choice?

I should add that Herbert Hoover was wildly successful at cutting government spending in the Great Depression. I am curious do you think this was successful for him? If not how is the same approach going to be successful today?

Errata Note: Original post referred to Ethiopia. The correct country should have been Somalia. I usually know my geography much better. However, since Somalia now has a government (as of December 2008), the last John Galt paradise is gone. Thanks to Peter for pointing this error out.

Posted in political, rants | 5 Comments »

Stations do NOT affect train speed

June 15th, 2009

Bullshit from an email thread and repeated constantly by the California High-Speed Rail Authority:

“Other potential benefits are also intriguing: a probable economic windfall for several cities along the route . . .”
and
“There will be as many as 24 passenger stations along the way . . .”

The train could more easily meet speed requirements if it stopped less. CA should do some land use planning first, then plan the train.

How would a station affect speed? Every train has to slow down to say “Hi”?

Stations have nothing to do with speed. Stations are INFRASTRUCTURE.

Station STOPS affect speed. Station STOPS are an OPERATIONAL decision and can be solved with a schedule change.

How come no one talks about having fewer freeway exits as a way to make freeways go faster? Because its stupid, drivers don’t get on and off at every freeway ramp. Yet somehow people think that a train has to behave like a stupid driver who takes every freeway exit! How come people think a train has to stop at every station?

News flash!

Trains can skip stations! If a train doesn’t stop a station the existence or (non-existence) of a station is irrelevant!

California High-Speed Rail Authority puts out this BULLSHIT as an excuse to avoid building a system that could actually serve dual purposes as both a long-distance system and a higher-speed adjunct to commuter rail.

“We can’t build more stations because that would slow down the train!”

This shinkansen train doesn’t look to be at all affected by the station:

Update 1:

Questions and astonishment from the email thread:

I’m not a train engineer, but I imagine money is spent on building a station with the expectation that some of the trains stop – because people live or work there.

I would hope so myself!

If there are stations for which trains stop infrequently, is that good planning?

Yes it is good planning. Stations can be built at relatively low cost. At the lowest end just a long enough siding, a asphalt boarding area, a parking lot and a place for taxis/buses. Even for HSR there is no reason a low frequency stop has to have even a building. Sure the HSR trains might require a high platform but that can be easily handled with a portable “step-up” carried on the train.

If I lived, or had a business near that station, I sure would want as many stops as possible.

well of course you would — and I want a pony. Wishes don’t mean you get.

Communities around those stations will use the existence of the station as marketing to attract development, but should it be happening at all 24 stops?

How is this a bad thing? Communities promoting passenger rail is bad?

Might there be some type of express/local arrangement, where a slower local (on a separate track) can feed the HSR? That doesn’t come through on the article.

You don’t need a separate track for anything except the station area. The station siding just needs to be long enough to allow for acceleration/deacceleration off of the main line. ( about 9 miles on the acceleration, less on the deacceleration ) Unless we are talking very impacted ROW the extra track is minor. The siding track is only necessary when the OPERATIONAL issues dictate that a train stopped at a station needs to be passed.

It probably doesn’t come through in the article because this is another operational issue that is decided after system is built. Running a train slower that system maximum because it is a “local” only matters when a higher speed express wants to pass. If there is nothing coming up behind it, a train can run at 40mph on the main high-speed line or it can stay stopped for 10 minutes blocking the main-line track.

How many stops do you think a typical train will make?

The system being proposed can run a train every 3 minutes / direction. 20 trains per hour going the direction you want as a passenger. If a passenger is going from one low traffic station to another they might have to transfer but at least they get service from a station close to their house. Why are we going to make them drive a long distance when the tracks are next to them. So what if only one train stops per day? This is just like Amtrak today!

Just make sure that the entire end-to-end trip time does not exceed a policy maximum.

How is it decided which stations to stop at?

By the people setting the schedule — same as today. Don’t get this question at all.

you’ve GOT to be kidding! Train Stations & Bus Stops have EVERYTHING to do with speed! Did you see: Muni floats plan to pull hundreds of S.F. stops

Really?
So the VTA 522 Rapid is slowed down by the existence of a bus stop on a curb? Did not notice that at all! Better go out and remove all those sign posts along El Camino Real for Route 22. That will definitely make the 522 go faster.

Update 2:

that bus is the transit agency’s version of BRT.
Not only have they eliminated stops (one mile length between most stop), it doesn’t even have a schedule after it leaves its origin at set time, meaning it doesn’t have to ‘wait’….

Amtrak buses doing something similar – on many runs, they will only stop if a passenger gets off, but not to pick up.

I think it important to recognize the differences between bus types – local, inter-city, BRT, and train types: commuter rail, heavy rail, LRT….speed is always an important factor – to some types more than others. Intercity trains are expected to go faster than commuter rail; commuter rail faster than heavy rail (though there can be express and local subway lines, of course), all both faster than LRT.

Can you imagine a ’smoke break’ for HSR???

And ?

Once again — how is the *existence* of a bus stop going to affect a bus’s schedule if the bus doesn’t stop at the stop?

How is the existence of a train station going to affect a given train’s speed and schedule if the train doesn’t stop?

I am completely baffled.

I am completely astonished that the NYC subway routes are impacted by stations where the train doesn’t stop.

I really do not understand how the “6-express” subway runs slower because of the existence of the “Elder Ave” stop ( which the 6-Express skips ). Perhaps someone else can help explain this?

I have no idea what a “smoke stop” has to do with station planning. Maybe you can enlighten me?

Update 3 [ 15 June 2009 23:13:00 ]:

An intelligent question from twitter:

Is there a (federal) law requiring trains to slow below a certain speed in stations irrespective of stopping?

No Federal Law. A train can go through at whatever speed it wants to. Certainly, some sort of warning system may desirable. The only exception is “holdout” stations. Holdout stations are stations where passengers cross active tracks to board. California Ave USED in Palo Alto, Ca to be holdout station.

Posted in environment, high-speed-rail, political, random silliness, rants, transportation | 2 Comments »

« Older Entries
Newer Entries »